GitHub issue: Dataset with streaming doesn't work with proxy
Author: YHL04 (NONE)
State: open | Created: 2024-06-22 16:12:08

### Describe the bug

I'm currently trying to stream data using dataset since the dataset is too big but it hangs indefinitely without loading the first batch. I use AIMOS which is a supercomputer that uses proxy to connect to the internet. I assume it has to do with the network configurations. I've already set up both HTTP_PROXY and HTTPS_PROXY. streaming = False works fine.

### Steps to reproduce the bug

use load_dataset with streaming = True in AIMOS

### Expected behavior

does not hang indefinitely and loads batches to start training run

### Environment info

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_pytorch_select           2.0                      cuda_2    https://ftp.osuosl.org/pub/open-ce/1.10.0
abseil-cpp                20220623.0           h9888cd1_6    conda-forge
absl-py                   1.0.0           py311h399429b_0    https://ftp.osuosl.org/pub/open-ce/1.10.0
aiofiles                  23.2.1             pyhd8ed1ab_0    conda-forge
aiohttp                   3.8.6           py311hf118e41_0
aiosignal                 1.2.0              pyhd3eb1b0_0
archspec                  0.2.3              pyhd8ed1ab_0    conda-forge
arrow-cpp                 11.0.0           ha3edaa6_5_cpu    conda-forge
async-timeout             4.0.2           py311h6ffa863_0
attrs                     23.1.0          py311h6ffa863_0
av                        10.0.0          py311he6153ed_2    https://ftp.osuosl.org/pub/open-ce/1.10.0
aws-c-auth                0.6.24               hb81f6d7_5    conda-forge
aws-c-cal                 0.5.20               h3c2b4d9_6    conda-forge
aws-c-common              0.8.11               h4194056_0    conda-forge
aws-c-compression         0.2.16               ha19333d_3    conda-forge
aws-c-event-stream        0.2.18               h12a9399_6    conda-forge
aws-c-http                0.7.4                ha2cde00_2    conda-forge
aws-c-io                  0.13.17              h9189062_2    conda-forge
aws-c-mqtt                0.8.6                h40d1a04_6    conda-forge
aws-c-s3                  0.2.4                hbdbe4f0_3    conda-forge
aws-c-sdkutils            0.1.7                ha19333d_3    conda-forge
aws-checksums             0.1.14               ha19333d_3    conda-forge
aws-crt-cpp               0.19.7               hd018011_7    conda-forge
aws-sdk-cpp               1.10.57              hb9575ba_4    conda-forge
blas                      1.0                    openblas
blinker                   1.8.2              pyhd8ed1ab_0    conda-forge
boltons                   23.0.0          py311h6ffa863_0
boost-cpp                 1.82.0               h25e6d66_2
bottleneck                1.3.5           py311h34f6284_0
brotli                    1.0.9                hf118e41_7
brotli-bin                1.0.9                hf118e41_7
brotli-python             1.0.9           py311h4a02239_7
bzip2                     1.0.8                h7b6447c_0
c-ares                    1.19.1               hf118e41_0
ca-certificates           2024.6.2             h0f6029e_0    conda-forge
cachetools                5.3.3              pyhd8ed1ab_0    conda-forge
certifi                   2024.6.2           pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py311hf118e41_3
charset-normalizer        2.0.4              pyhd3eb1b0_0
click                     8.1.7           unix_pyh707e725_0    conda-forge
conda                     24.5.0          py311h1af927a_0    conda-forge
conda-content-trust       0.2.0           py311h6ffa863_0
conda-libmamba-solver     23.11.1         py311h6ffa863_0
conda-package-handling    2.2.0           py311h6ffa863_0
conda-package-streaming   0.9.0           py311h6ffa863_0
contourpy                 1.0.5           py311h25e6d66_0
cryptography              41.0.3          py311hb0e80e7_0
cudatoolkit               11.8.0              hedcfb66_13    conda-forge
cudnn                     8.9.2_11.8           h9ceb136_1    https://ftp.osuosl.org/pub/open-ce/1.10.0
cycler                    0.11.0             pyhd3eb1b0_0
datasets                  2.12.0          py311h6ffa863_0
dill                      0.3.6           py311h6ffa863_0
distro                    1.9.0              pyhd8ed1ab_0    conda-forge
ffmpeg                    4.2.2                  opence_0    https://ftp.osuosl.org/pub/open-ce/1.10.0
filelock                  3.9.0           py311h6ffa863_0
fmt                       9.1.0                h25e6d66_0
fonttools                 4.25.0             pyhd3eb1b0_0
freetype                  2.12.1               hd23a775_0
frozendict                2.4.4           py311hb02d432_0    conda-forge
frozenlist                1.4.0           py311hf118e41_0
fsspec                    2023.9.2        py311h6ffa863_0
gflags                    2.2.2                he6710b0_0
giflib                    5.2.1                hf118e41_3
glog                      0.6.0                hbe088e0_0    conda-forge
gmp                       6.3.0                h46f38da_0    conda-forge
gmpy2                     2.1.5           py311h2758da7_1    conda-forge
google-auth               2.30.0             pyhff2d567_0    conda-forge
google-auth-oauthlib      0.5.3              pyhd8ed1ab_0    conda-forge
grpc-cpp                  1.51.1               h8ba971d_1    conda-forge
grpcio                    1.54.3          py311h414e0d3_0    https://ftp.osuosl.org/pub/open-ce/1.10.0
huggingface_hub           0.17.3          py311h6ffa863_0
icu                       73.1                 h4a02239_0
idna                      3.4             py311h6ffa863_0
importlib-metadata        6.0.0           py311h6ffa863_0
jinja2                    3.1.4              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   hf118e41_1
jsonpatch                 1.32               pyhd3eb1b0_0
jsonpointer               2.1                pyhd3eb1b0_0
kiwisolver                1.4.4           py311h4a02239_0
krb5                      1.20.1               hc019ccd_1
lame                      3.100             hb283c62_1003    conda-forge
lcms2                     2.12                 h2045e0b_0
ld_impl_linux-ppc64le     2.38                 hec883e6_1
lerc                      3.0                  h29c3540_0
leveldb                   1.23                 h24532b4_1    conda-forge
libabseil                 20220623.0      cxx17_h9235812_6    conda-forge
libarchive                3.6.2                hd8ab008_2
libarrow                  11.0.0           h837770b_5_cpu    conda-forge
libboost                  1.82.0               haf51a6a_2
libbrotlicommon           1.0.9                hf118e41_7
libbrotlidec              1.0.9                hf118e41_7
libbrotlienc              1.0.9                hf118e41_7
libcrc32c                 1.1.2                h3b9df90_0    conda-forge
libcurl                   8.4.0                h4d62439_0
libdeflate                1.17                 hf118e41_1
libedit                   3.1.20221030         hf118e41_0
libev                     4.33                 h140841e_1
libevent                  2.1.10               h19c23f1_4    conda-forge
libexpat                  2.6.2                h46f38da_0    conda-forge
libffi                    3.4.4                h4a02239_0
libgcc-ng                 13.2.0              h31e42bb_10    conda-forge
libgfortran-ng            11.2.0               hb3889a9_1
libgfortran5              11.2.0               h1234567_1
libgomp                   13.2.0              h31e42bb_10    conda-forge
libgoogle-cloud           2.7.0                h11140b6_1    conda-forge
libgrpc                   1.51.1               h4d29a31_1    conda-forge
libmamba                  1.5.3                h7c6fafd_0
libmambapy                1.5.3           py311h828bf7b_0
libnghttp2                1.57.0               h44e5816_0
libnsl                    2.0.1                ha17a0cc_0    conda-forge
libopenblas               0.3.23               hc5a31fb_2    https://ftp.osuosl.org/pub/open-ce/1.10.0
libopus                   1.3.1                h4e0d66e_1    conda-forge
libpng                    1.6.39               hf118e41_0
libprotobuf               3.21.12              h1776448_0    https://ftp.osuosl.org/pub/open-ce/1.10.0
libsolv                   0.7.24               h0f529ac_0
libsqlite                 3.45.3               hd4bbf49_0    conda-forge
libssh2                   1.10.0               h50fa78f_2
libstdcxx-ng              13.2.0              h262982c_10    conda-forge
libthrift                 0.18.0               h82f1162_0    conda-forge
libtiff                   4.5.1                h4a02239_0
libutf8proc               2.8.0                hb283c62_0    conda-forge
libuuid                   2.38.1               h4194056_0    conda-forge
libvpx                    1.13.1               h46f38da_0    conda-forge
libwebp                   1.3.2                h0f96ee2_0
libwebp-base              1.3.2                hf118e41_0
libxcrypt                 4.4.36               ha17a0cc_1    conda-forge
libxml2                   2.10.4               h18e3229_1
libzlib                   1.2.13               h1f2b957_6    conda-forge
llvm-openmp               14.0.6               hc028133_0    https://ftp.osuosl.org/pub/open-ce/1.10.0
lmdb                      0.9.31               ha17a0cc_1    conda-forge
lz4-c                     1.9.4                h4a02239_0
markdown                  3.4.4              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.5           py311h32d8acf_0    conda-forge
matplotlib                3.8.0           py311h6ffa863_0
matplotlib-base           3.8.0           py311h52e1fcc_0
menuinst                  2.1.1           py311h1af927a_0    conda-forge
mpc                       1.3.1                heaf1863_0    conda-forge
mpfr                      4.2.1                haad2271_1    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
multidict                 6.0.2           py311hf118e41_0
multiprocess              0.70.14         py311h6ffa863_0
munkres                   1.1.4                      py_0
mypy_extensions           1.0.0              pyha770c72_0    conda-forge
nccl                      2.18.3               cuda11.8_1    https://ftp.osuosl.org/pub/open-ce/1.10.0
ncurses                   6.4                  h4a02239_0
nest-asyncio              1.6.0              pyhd8ed1ab_0    conda-forge
networkx                  2.8.8              pyhd8ed1ab_0    conda-forge
nomkl                     3.0                           0    https://ftp.osuosl.org/pub/open-ce/1.10.0
numactl                   2.0.16               hba61f60_1    https://ftp.osuosl.org/pub/open-ce/1.10.0
numexpr                   2.8.7           py311hc46fc55_0
numpy                     1.24.3          py311h148a09e_0
numpy-base                1.24.3          py311h06b82f6_0
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
openjpeg                  2.4.0                hfe35807_0
openssl                   3.3.1                h1f2b957_0    conda-forge
orc                       1.8.2                h341c9a4_2    conda-forge
packaging                 23.1            py311h6ffa863_0
pandas                    2.1.1           py311h52e1fcc_0
pcre2                     10.42                h280155c_0
pillow                    10.0.1          py311he33076b_0
pip                       23.3            py311h6ffa863_0
platformdirs              4.2.2              pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0           py311h6ffa863_1
pooch                     1.8.2              pyhd8ed1ab_0    conda-forge
protobuf                  4.21.12         py311ha7baec7_1    https://ftp.osuosl.org/pub/open-ce/1.10.0
psutil                    5.9.8           py311hd26027c_0    conda-forge
pyarrow                   11.0.0          py311h04a18d5_1
pyasn1                    0.6.0              pyhd8ed1ab_0    conda-forge
pyasn1-modules            0.4.0              pyhd8ed1ab_0    conda-forge
pybind11-abi              4                    hd3eb1b0_1
pycosat                   0.6.6           py311hf118e41_0
pycparser                 2.21               pyhd3eb1b0_0
pyjwt                     2.8.0              pyhd8ed1ab_1    conda-forge
pyopenssl                 23.2.0          py311h6ffa863_0
pyparsing                 3.0.9           py311h6ffa863_0
pyre-extensions           0.0.30             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1           py311h6ffa863_0
python                    3.11.8          h3332dee_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd3eb1b0_0
python-tzdata             2023.3             pyhd3eb1b0_0
python-xxhash             2.0.2           py311hf118e41_1
python_abi                3.11                    4_cp311    conda-forge
pytorch                   2.0.1           cuda11.8_py311_1    https://ftp.osuosl.org/pub/open-ce/1.10.0
pytorch-base              2.0.1           cuda11.8_py311_pb4.21.12_4    https://ftp.osuosl.org/pub/open-ce/1.10.0
pytz                      2023.3.post1    py311h6ffa863_0
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.1           py311hf118e41_0
re2                       2023.02.01           h883269e_0    conda-forge
readline                  8.2                  hf118e41_0
regex                     2023.10.3       py311hf118e41_0
reproc                    14.2.4               h29c3540_1
reproc-cpp                14.2.4               h29c3540_1
requests                  2.31.0          py311h6ffa863_0
requests-oauthlib         2.0.0              pyhd8ed1ab_0    conda-forge
responses                 0.13.3             pyhd3eb1b0_0
rsa                       4.9                pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.17.21         py311hf118e41_0
s2n                       1.3.37               h5e47323_0    conda-forge
safetensors               0.4.0           py311hda16d9e_0
scipy                     1.11.1          py311hd69e9bb_0    https://ftp.osuosl.org/pub/open-ce/1.10.0
sentencepiece             0.1.97          h1e74c73_py311_pb4.21.12_2    https://ftp.osuosl.org/pub/open-ce/1.10.0
setuptools                68.0.0          py311h6ffa863_0
six                       1.16.0             pyhd3eb1b0_1
snappy                    1.1.9                h29c3540_0
sqlite                    3.41.2               hf118e41_0
sympy                     1.12.1          pypyh2585a3b_103    conda-forge
tabulate                  0.8.10             pyhd8ed1ab_0    conda-forge
tensorboard               2.13.0          pyhab0730d_pb4.21.12_1    https://ftp.osuosl.org/pub/open-ce/1.10.0
tensorboard-data-server   0.7.0              pyh6f84499_1    https://ftp.osuosl.org/pub/open-ce/1.10.0
tensorboard-plugin-wit    1.6.0              pyh9f0ad1d_0    conda-forge
tk                        8.6.13               hd4bbf49_0    conda-forge
tokenizers                0.13.3          py311h3d4f45a_0
torchdata                 0.6.0                   py311_2    https://ftp.osuosl.org/pub/open-ce/1.10.0
torchsnapshot             0.1.0              pyhd8ed1ab_0    conda-forge
torchtext-base            0.15.2          cuda11.8_py311_1    https://ftp.osuosl.org/pub/open-ce/1.10.0
torchtnt                  0.2.4              pyhd8ed1ab_0    conda-forge
torchvision-base          0.15.2          cuda11.8_py311_1    https://ftp.osuosl.org/pub/open-ce/1.10.0
tornado                   6.3.3           py311hf118e41_0
tqdm                      4.65.0          py311h7837921_0
transformers              4.32.1          py311h6ffa863_0
truststore                0.8.0           py311h6ffa863_0
typing-extensions         4.7.1           py311h6ffa863_0
typing_extensions         4.7.1           py311h6ffa863_0
typing_inspect            0.9.0              pyhd8ed1ab_0    conda-forge
tzdata                    2023c                h04d1e81_0
urllib3                   1.26.18         py311h6ffa863_0
utf8proc                  2.6.1                h140841e_0
werkzeug                  2.3.8              pyhd8ed1ab_0    conda-forge
wheel                     0.41.2          py311h6ffa863_0
xxhash                    0.8.0                h140841e_3
xz                        5.4.2                hf118e41_0
yaml                      0.2.5                h7b6447c_0
yaml-cpp                  0.8.0                h4a02239_0
yarl                      1.8.1           py311hf118e41_0
zipp                      3.11.0          py311h6ffa863_0
zlib                      1.2.13               h1f2b957_6    conda-forge
zstandard                 0.19.0          py311hf118e41_0
zstd                      1.5.5                h57e4825_0