Support proxy and user-agent in fsspec calls
State: open
Labels: description: New feature or request
name: enhancement

description: Issues a bit more difficult than "Good First" issues
name: good second issue

Since we switched to the new HfFileSystem we no longer apply user's proxy and user-agent.

Using the HTTP_PROXY and HTTPS_PROXY environment variables works though since we use aiohttp to call the HF Hub.

This can be implemented in `_prepare_single_hop_path_and_storage_options`.

Though ideally the `HfFileSystem` could support passing at least the proxies

Comments:
hii @lhoestq can you assign this issue to me?


You can reply "#self-assign" to this issue to automatically get assigned to it :)
Let me know if you have any questions or if I can help

#2289 

Actually i am quite new to figure it out how everything goes and done 

> You can reply "#self-assign" to this issue to automatically get assigned to it :)
> Let me know if you have any questions or if I can help

when i wrote #self-assign it automatically got converted to some number is it correct or i have done it some wrong way, I am quite new to open source thus wanna try to learn and explore it

#2289 #self-assign 

Ah yea github tries to replace the #self-assign with an issue link. I guess you can try to copy-paste instead to see if it works

Anyway let me assign you manually

thanks a lot @lhoestq ! though i have a very lil idea of the issue, i am new. as i said before,  but gonna try my best shot to do it.
can you please suggest some tips or anything from your side, how basically we approach it will be really helpfull.
Will try my best!

The HfFileSystem from the `huggingface_hub` package can already read the HTTP_PROXY and HTTPS_PROXY environment variables. So the remaining thing missing is the `user_agent` that the user may include in a `DownloadConfig` object. The user agent can be used for regular http calls but also calls to the HfFileSystem.

- for http, the `user_agent` isn't passed from `DownloadConfig` to `get_datasets_user_agent` in `_prepare_single_hop_path_and_storage_options` in `streaming_download_manager.py` so we need to include it
- for HfFileSystem I think it requires a PR in https://github.com/huggingface/huggingface_hub to include it in the `HfFileSystem.__init__`

Hi @lhoestq 👋🏼

Is anyone currently working on this? If not, I'd like to pick it up.

As I understand it:
- The `user_agent` from `DownloadConfig` isn't currently passed to `get_datasets_user_agent()` inside `_prepare_single_hop_path_and_storage_options`.
- I'll update that function to include the correct `user-agent` in the `headers`.
- For full support, we may also need a change in `huggingface_hub` to let `HfFileSystem` accept custom headers.

Please let me know if this approach sounds good or if you’d prefer it handled differently 🙌


Hi @lhoestq 👋🏼 Just following up on this one!

I’ve opened [PR #7631](https://github.com/huggingface/datasets/pull/7631) to pass the user_agent from DownloadConfig into the fsspec storage options via _prepare_single_hop_path_and_storage_options (for HTTP/S).

For full support in HfFileSystem, I understand we might need a corresponding PR in huggingface_hub to add user_agent handling in its __init__. If you're okay with that direction, I can also raise a PR there!

Let me know if any adjustments are needed 🙌