opened
JkSelf
username_0
## What changes were proposed in this pull request?
In Tencent cloud environment, when enable ae, it will convert smj to bhj in runtime based on the compressed size of small table with 16M. However, when building hash table, the decompressed size of small table can be up to 2GB and the row counts can be 134485048, which cause the OOM. In order to resolve this issue, this PR change the condition from compressed size to row counts when converting smj to bhj.

## How was this patch tested?
Existing tests.
Change the condition of converting smj to bhj from compressed size to row counts
issue
created
JkSelf
username_0
@jerrychenhf @username_1 @plusplusjiajia  Please help me review if you have available time. Thanks for your help.
comment
created
JkSelf
username_0
@username_1 @plusplusjiajia This PR has complete the verification in Tencent cloud environment. Please help me review. Thanks.
comment
created
carsonwang
username_1
@username_0 , there is similar work in #69 which tries to get the accurate data size from metrics. However it is still pending on another Spark PR to make sure the metrics are not updated twice.
comment
created
JkSelf
username_0
@username_1  The issue to resolve is similar. And the implement method may be not similar, here we estimate the decompressed size by the row count * data type. And the row count is collected in `ShuffleWrite` based on the real data size record, and does not have the updated twice issue I think.
comment
created
carsonwang
username_1
Yes, I was thinking the size we get from metrics are more accurate except for the potential multiple update issue. The calculation in this PR is an estimation because for type like String, we don't know the exact size.
comment
created
JkSelf
username_0
@username_1 Yes. Even we get the accurate `dataSize` from the ExchangeExec. We may still need to estimate the data size of other operators (ProjectExec or FilterExec) based on the data type in `SizeInBytesOnlyStatsPlanVisitor` class, Right?
comment