HSTU large model loading using in-tensor multi-threading
Labels: caffe2, fb-exported, topic: not user facing

Summary:
Zion-4s core has poor perf when it comes to reading the large tensor (e.g. 300G), no matter for manifold downloading or reading from files. In this diff, I changed the getRecord function from single thread to multiple threads by passing multiple readers to getRecord function and access the same record at different chunks with different readers.
We control the number of additional reader with the`sigrid_model_manager_additional_reader` flag. The default value is 0. When `additional_reader=2`, we allocate `2` extra read client threads.

Profiler Results:
**multi readers**
ods counter:
https://fburl.com/ods/s12q1t6j
strobelight:
https://fburl.com/strobelight/r1jj17qs
https://fburl.com/strobelight/702j72k8
download:
https://fburl.com/scuba/download_client_stats/5fh7nlu7
 {F1118644555}
**single reader**
ods counter:
https://fburl.com/ods/m9sv2sp0
strobelight:
https://fburl.com/strobelight/emc9q6z0
https://fburl.com/strobelight/mn23ctda
download:
https://fburl.com/scuba/download_client_stats/xsl3pq8y
 {F1118644696}

Test Plan:
**unit test**
```
buck2 run @//mode/dev //caffe2/caffe2/serialize:inline_container_test
```

**Local model loading**
```
SIGRID_ADDITIONAL_READER=4 SIGRID_MAX_RSS_SIZE_BYTES=751619276800 PYTORCH_PREDICTOR_ENABLE_XL_FORMAT_V2=true PYTORCH_PREDICTOR_ENABLE_XL_FORMAT_V2_INPLACE_LOADING=true GIF_LOAD_LOCAL_NET=true RUN_D2H_TOGETHER_WITH_EXECUTION_STREAM=true EVENT_POOL_ENABLE_BLOCKING_SYNC=false ENABLE_DEPLOY_INPLACE_LOADING=true TGIF_REPLICATE_MERGE_BY_TEMPFILE=true USE_STATIC_PATH=1 USE_ALL_TO_ONE_OP=false MAX_NUM_ADS=10240 REQUEST_BATCHING_PARAM_OVERRIDE="max_batch_size|${MAX_NUM_ADS};batch_time_us|50000" NUM_DEPLOY_INTERPRETER=32 NUM_DESER_AND_REMOTE_RO_CPU_WORKERS=20 MODULE_NUM_WORKERS_PER_GPU='merge|8;remote|0' MODEL_ID=962580335 SNAPSHOT_ID=0 SERVER_PORT=7456 CUDA_VISIBLE_DEVICES_FOR_PREDICTOR="4,5,6,7" CPU_NUMA_NODES_FOR_PREDICTOR="2,3" ENABLE_THRIFT_WARMUP=false hpc/inference/scripts/gif/prospector/1_cards/launch_gpu_sigrid_predictor_task_0.sh 2>&1 | tee predictor.txt
```
Load from local: P850152856
Load from manifold: P850716434

**Vanguard model loading**
```
VANGUARD_ENV=dev vg serving-test -m 959387101 -s 0 --spec ~/fbsource/fbcode/feed_ranking_infra/vanguard/specs/umia_hstu_dev_xlv2.yaml 2>&1 | tee tmp.txt
```
1TB model 959387101
baseline: 25min, P844194768
pkg a44e273: 9min, P848107231


Prod model 960881812
baseline: 15min
pkg a44e273: 5min, P850724551

**Inference eval**
```
VANGUARD_ENV=dev vg predictor launch -m 960881812 -s 602 --spec ~/fbsource/fbcode/feed_ranking_infra/vanguard/specs/umia_hstu_dev_xlv2.yaml 2>&1 | tee tmp.txt
pkg_version: a44e273

## mvai_umia_hstu_fbr:recurring_eval
hg checkout 1df2db0a09e755e80e7b3a7943f49c1f5e71567b
light -d ~/fbsource/fbcode/minimal_viable_ai -d ~/fbsource/fbcode/hpc/ ~/fbsource/fbcode/minimal_viable_ai/sandbox/eval/predictor_eval.py --model-entity-id 960881812 --snapshot 602 --eval-config umia/fbr_hstu_gif_ts_sampling_diversity_disable_both --tier feed.sigrid.predictor.benchmark.vg_tissue030_dev --num-requests 100000 --eval-ts 2023-10-10+11:00:99 2>&1 | tee eval.txt
```
baseline: https://fburl.com/mlhub/yvmx9ald
> I1010 09:11:52.131 4846 predictor_eval.py:558 [Rank 0] ## Eval Results: {"recall@1/vvp100": 0.013800769113004208, "recall@5/vvp100": 0.049447201192379, "recall@20/vvp100": 0.11355383694171906, "recall@100/vvp100": 0.17039309442043304, "recall@200/vvp100": 0.25405290722846985, "recall@500/vvp100": 0.4610816240310669, "recall@1000/vvp100": 0.637019693851471, "recall@1250/vvp100": 0.6741645932197571, "num_interactions/vvp100": 83826.0}

pkg: a44e273 with multi-thread: P850634507

> I1010 11:15:36.873 4038353 predictor_eval.py:558 [Rank 0] ## Eval Results: {"recall@1/vvp100": 0.01417488232254982, "recall@5/vvp100": 0.050109997391700745, "recall@20/vvp100": 0.1138257384300232, "recall@100/vvp100": 0.17026753723621368, "recall@200/vvp100": 0.25343313813209534, "recall@500/vvp100": 0.4619934856891632, "recall@1000/vvp100": 0.6371392011642456, "recall@1250/vvp100": 0.6746780872344971, "num_interactions/vvp100": 93684.0}

Differential Revision: D49969050