Repository: beetbox/beets

Issue:
fetchart: Logged requests ignore proxy settings from environment variables
### Problem
I recently noticed that beet is not embedding album art in my music files as I import them if they are not available locally. So I ran`beet fetchart` command and tried running it in verbose mode for one of the failed albums.

I think I should mention that I am using internet behind a proxy server, not sure if this has anything to do with the error but I am suspicious that its the proxy.

I first ran the import command in verbose mode which gave the following output:

```sh
slimshady@Slim-Shady:~/Downloads/Unfixed/Céline Dion$ beet -v import .
user configuration: /home/slimshady/.config/beets/config.yaml
data directory: /home/slimshady/.config/beets
plugin paths: /home/slimshady/Apps/beets/plugins
Sending event: pluginload
library database: /home/slimshady/Apps/beets/musiclibrary.db
library directory: /home/slimshady/Music
Sending event: library_opened
Sending event: import_begin
Sending event: import_task_created
Sending event: import_task_start
Looking up: /home/slimshady/Downloads/Unfixed/Céline Dion
Tagging  - 
No album ID found.
Search terms:  - 
Album might be VA: True
Sending event: albuminfo_received
Candidate: 423kidk, Lil Somethin' - Kyle (2771484)
Computing track assignment...
...done.
Success. Distance: 0.91
Sending event: albuminfo_received
Candidate: TRiNoVaNTe - Love for You (2647161)
Computing track assignment...
...done.
Success. Distance: 0.90
Sending event: albuminfo_received
Candidate: Figub Brazlevic - Kaseta De Ouf (2785162)
Computing track assignment...
...done.
Success. Distance: 0.96
Sending event: albuminfo_received
Candidate: Azetune - OXT (2729859)
Computing track assignment...
...done.
Success. Distance: 0.88
Sending event: albuminfo_received
Candidate: Namzo - Kensley Circuit (2689278)
Computing track assignment...
...done.
Success. Distance: 0.89
Requesting MusicBrainz release fa6a5cfc-8418-4644-a99c-3180821ca360
primary MB release type: album
secondary MB release type(s): compilation, soundtrack
Sending event: albuminfo_received
Requesting MusicBrainz release 2f347cfb-eab9-41da-ab7f-8b4d512a6c15
primary MB release type: album
secondary MB release type(s): compilation
Sending event: albuminfo_received
Requesting MusicBrainz release bfd873a1-d81b-46f5-90ba-0a93ce2c8ffe
primary MB release type: album
secondary MB release type(s): compilation, soundtrack
Sending event: albuminfo_received
Requesting MusicBrainz release f3c7e885-e459-43e5-865c-95c62a9781b9
primary MB release type: album
secondary MB release type(s): compilation, soundtrack
Sending event: albuminfo_received
chroma: acoustid album candidates: 4
Sending event: albuminfo_received
Candidate: Various Artists - Deadpool 2: Original Motion Picture Soundtrack (fa6a5cfc-8418-4644-a99c-3180821ca360)
Computing track assignment...
...done.
Success. Distance: 0.93
Sending event: albuminfo_received
Candidate: Various Artists - Kuschelrock 32 (2f347cfb-eab9-41da-ab7f-8b4d512a6c15)
Computing track assignment...
...done.
Success. Distance: 0.97
Sending event: albuminfo_received
Candidate: Various Artists - Deadpool 2 (Super Duper Cut) (bfd873a1-d81b-46f5-90ba-0a93ce2c8ffe)
Computing track assignment...
...done.
Success. Distance: 0.95
Sending event: albuminfo_received
Candidate: Various Artists - Deadpool 2: Original Motion Picture Soundtrack (f3c7e885-e459-43e5-865c-95c62a9781b9)
Computing track assignment...
...done.
Success. Distance: 0.93
Evaluating 9 candidates.

/home/slimshady/Downloads/Unfixed/Céline Dion (1 items)
Sending event: before_choose_candidate
Finding tags for album " - ".
Candidates:
1. Azetune - OXT (11.7%) (album, artist, tracks, ...) (Beatport, Digital, 2020, YouTunez, 4250693219135)
2. Namzo - Kensley Circuit (10.7%) (album, artist, tracks, ...) (Beatport, Digital, 2020, Karia Records, KR037)
3. TRiNoVaNTe - Love for You (9.9%) (album, artist, tracks, ...) (Beatport, Digital, 2020, KP Recordings, KP462)
4. 423kidk, Lil Somethin' - Kyle (9.2%) (album, artist, missing tracks, ...) (Beatport, Digital, 2020, SpacedOut Studios Entertainment, KL12062019)
5. Various Artists - Deadpool 2: Original Motion Picture Soundtrack (7.3%) (missing tracks, album, tracks) (CD, 2018, US, Columbia, 19075853472)
6. Various Artists - Deadpool 2: Original Motion Picture Soundtrack (7.3%) (missing tracks, album, tracks) (Digital Media, 2018, GB, Columbia)
7. Various Artists - Deadpool 2 (Super Duper Cut) (5.4%) (missing tracks, album, tracks) (Digital Media, 2018, US, Columbia)
8. Figub Brazlevic - Kaseta De Ouf (4.1%) (missing tracks, album, artist, ...) (Beatport, Digital, 2020, VinDig, VINDIG432)
9. Various Artists - Kuschelrock 32 (2.5%) (missing tracks, album, tracks) (2xCD, 2018, XE, Sony Music, 19075 86604 2)
# selection (default 1), Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort? 6
Correcting tags from:
    (unknown album)
To:
    Deadpool 2: Original Motion Picture Soundtrack
URL:
    https://musicbrainz.org/release/f3c7e885-e459-43e5-865c-95c62a9781b9
(Similarity: 7.3%) (missing tracks, album, tracks) (Digital Media, 2018, GB, Columbia)
 * Céline Dion - Ashes.mp3 (#0) (3:52) -> Ashes (#1) (3:19) (title, length)
Missing tracks (11/12 - 91.7%):
 ! Welcome to the Party                         (# 2) (3:01)
 ! Nobody Speak                                 (# 3) (3:16)
 ! In Your Eyes                                 (# 4) (5:29)
 ! Take on Me (MTV Unplugged – Summer Solstice) (# 5) (4:14)
 ! If I Could Turn Back Time                    (# 6) (4:01)
 ! 9 to 5                                       (# 7) (2:45)
 ! All Out of Love                              (# 8) (4:03)
 ! We Belong (single version)                   (# 9) (3:42)
 ! Tomorrow                                     (#10) (2:30)
 ! Mutant Convoy                                (#11) (3:59)
 ! Bangarang                                    (#12) (3:35)
Apply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort? a
Sending event: import_task_choice
Sending event: import_task_apply
0 of 1 items replaced
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
lyrics: fetched lyrics: Céline Dion - Deadpool 2: Original Motion Picture Soundtrack - Ashes
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: item_copied
Sending event: database_change
Sending event: database_change
Sending event: write
Sending event: after_write
Sending event: database_change
Sending event: import_task_files
acousticbrainz: getting data for: Céline Dion - Deadpool 2: Original Motion Picture Soundtrack - Ashes
Sending event: database_change
Sending event: album_imported
Sending event: import
Sending event: cli_exit
```
Them running fetchart command in verbose (`-v`) mode:

```sh
$ beet -v fetchart Deadpool\ 2
```

Gave the output as follows:

```sh
slimshady@Slim-Shady:~/Downloads/Unfixed/Céline Dion$ beet -v fetchart Deadpool\ 2
user configuration: /home/slimshady/.config/beets/config.yaml
data directory: /home/slimshady/.config/beets
plugin paths: /home/slimshady/Apps/beets/plugins
Sending event: pluginload
library database: /home/slimshady/Apps/beets/musiclibrary.db
library directory: /home/slimshady/Music
Sending event: library_opened
fetchart: trying source filesystem for album Various Artists - Deadpool 2: Original Motion Picture Soundtrack
fetchart: trying source itunes for album Various Artists - Deadpool 2: Original Motion Picture Soundtrack
fetchart: getting URL: https://itunes.apple.com/search?term=Various+Artists+Deadpool+2%3A+Original+Motion+Picture+Soundtrack&entity=album&media=music&limit=200
fetchart: iTunes search failed: HTTPSConnectionPool(host='itunes.apple.com', port=443): Max retries exceeded with url: /search?term=Various+Artists+Deadpool+2%3A+Original+Motion+Picture+Soundtrack&entity=album&media=music&limit=200 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f5b2494bac8>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
fetchart: trying source coverart for album Various Artists - Deadpool 2: Original Motion Picture Soundtrack
fetchart: downloading image: https://coverartarchive.org/release/f3c7e885-e459-43e5-865c-95c62a9781b9/front
fetchart: error fetching art: HTTPSConnectionPool(host='coverartarchive.org', port=443): Max retries exceeded with url: /release/f3c7e885-e459-43e5-865c-95c62a9781b9/front (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f5b2494b978>: Failed to establish a new connection: [Errno 110] Connection timed out',))
fetchart: trying source coverart for album Various Artists - Deadpool 2: Original Motion Picture Soundtrack
fetchart: downloading image: https://coverartarchive.org/release-group/22a7ab97-a014-41f2-96c7-98df794e01bb/front
fetchart: error fetching art: HTTPSConnectionPool(host='coverartarchive.org', port=443): Max retries exceeded with url: /release-group/22a7ab97-a014-41f2-96c7-98df794e01bb/front (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f5b2494b0f0>: Failed to establish a new connection: [Errno 110] Connection timed out',))
fetchart: trying source amazon for album Various Artists - Deadpool 2: Original Motion Picture Soundtrack
fetchart: trying source albumart for album Various Artists - Deadpool 2: Original Motion Picture Soundtrack
fetchart: trying source wikipedia for album Various Artists - Deadpool 2: Original Motion Picture Soundtrack
fetchart: getting URL: https://dbpedia.org/sparql?format=application%2Fsparql-results%2Bjson&timeout=2500&query=PREFIX+rdf%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F1999%2F02%2F22-rdf-syntax-ns%23%3E%0A+++++++++++++++++PREFIX+dbpprop%3A+%3Chttp%3A%2F%2Fdbpedia.org%2Fproperty%2F%3E%0A+++++++++++++++++PREFIX+owl%3A+%3Chttp%3A%2F%2Fdbpedia.org%2Fontology%2F%3E%0A+++++++++++++++++PREFIX+rdfs%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3E%0A+++++++++++++++++PREFIX+foaf%3A+%3Chttp%3A%2F%2Fxmlns.com%2Ffoaf%2F0.1%2F%3E%0A%0A+++++++++++++++++SELECT+DISTINCT+%3FpageId+%3FcoverFilename+WHERE+%7B%0A+++++++++++++++++++%3Fsubject+owl%3AwikiPageID+%3FpageId+.%0A+++++++++++++++++++%3Fsubject+dbpprop%3Aname+%3Fname+.%0A+++++++++++++++++++%3Fsubject+rdfs%3Alabel+%3Flabel+.%0A+++++++++++++++++++%7B+%3Fsubject+dbpprop%3Aartist+%3Fartist+%7D%0A+++++++++++++++++++++UNION%0A+++++++++++++++++++%7B+%3Fsubject+owl%3Aartist+%3Fartist+%7D%0A+++++++++++++++++++%7B+%3Fartist+foaf%3Aname+%22Various+Artists%22%40en+%7D%0A+++++++++++++++++++++UNION%0A+++++++++++++++++++%7B+%3Fartist+dbpprop%3Aname+%22Various+Artists%22%40en+%7D%0A+++++++++++++++++++%3Fsubject+rdf%3Atype+%3Chttp%3A%2F%2Fdbpedia.org%2Fontology%2FAlbum%3E+.%0A+++++++++++++++++++%3Fsubject+dbpprop%3Acover+%3FcoverFilename+.%0A+++++++++++++++++++FILTER+%28+regex%28%3Fname%2C+%22Deadpool+2%3A+Original+Motion+Picture+Soundtrack%22%2C+%22i%22%29+%29%0A++++++++++++++++++%7D%0A+++++++++++++++++Limit+1
fetchart: dbpedia: error receiving response
fetchart: trying source fanarttv for album Various Artists - Deadpool 2: Original Motion Picture Soundtrack
fetchart: getting URL: https://webservice.fanart.tv/v3/music/albums/22a7ab97-a014-41f2-96c7-98df794e01bb
fetchart: fanart.tv: error receiving response
fetchart: Various Artists - Deadpool 2: Original Motion Picture Soundtrack: no art found
Sending event: cli_exit
```
### Setup

* OS: Ubuntu 18.04
* Python version:  3.6.9
* beets version: 1.4.9

My configuration (output of `beet config`) is:

```yaml
lyrics:
    bing_lang_from: []
    auto: yes
    force: yes
    sources: google lyricwiki musixmatch genius
    google_API_key: REDACTED
    bing_client_secret: REDACTED
    bing_lang_to:
    google_engine_ID: REDACTED
    genius_api_key: REDACTED
    fallback:
    local: no
directory: ~/Music
library: ~/Apps/beets/musiclibrary.db

import:
    write: yes
    log: ~/Apps/beets/import-log.txt

match:
    strong_rec_thresh: 0.15
    medium_rec_thresh: 0.3

plugins: acousticbrainz chroma fetchart lyrics embedart lastgenre ftintitle beatport
pluginpath: ~/Apps/beets/plugins

paths:
    default: $albumartist/$album%aunique{}/$track - $title
    singleton: $artist/Singles/$title
acousticbrainz:
    auto: yes
    force: yes
    tags: []
chroma:
    auto: yes
fetchart:
    auto: yes
    sources: filesystem itunes *
    minwidth: 0
    maxwidth: 0
    enforce_ratio: no
    cautious: no
    cover_names:
    - cover
    - front
    - art
    - album
    - folder
    google_key: REDACTED
    google_engine: 001442825323518660753:hrh5ch1gjzm
    fanarttv_key: REDACTED
    store_source: no
embedart:
    auto: yes
    maxwidth: 0
    compare_threshold: 0
    ifempty: no
    remove_art_file: no
lastgenre:
    auto: yes
    force: yes
    whitelist: yes
    min_weight: 10
    count: 1
    fallback:
    canonical: no
    source: album
    separator: ', '
    prefer_specific: no
ftintitle:
    auto: yes
    format: feat. {0}
    drop: no
beatport:
    apikey: REDACTED
    apisecret: REDACTED
    tokenfile: beatport_token.json
    source_weight: 0.5
```



Discussion:
Are you able to reach the internet from the command line on the machine you're trying this on? Do you need a proxy server to connect to the internet (e.g. in a corporate environment)?

What's the output from this command?

```Text
curl -v -o /dev/null https://ia803109.us.archive.org/33/items/mbid-f3c7e885-e459-43e5-865c-95c62a9781b9/mbid-f3c7e885-e459-43e5-865c-95c62a9781b9-19835076126.jpg
```
Yes, I am able to reach the internet from the command line.
Yes, I do need proxy to connect to the internet, it is my university's proxy server.  My proxy environment variables are set too.

The output for the above mentioned command is: 

```sh
slimshady@Slim-Shady:~/Downloads$ curl -v -o /dev/null https://ia803109.us.archive.org/33/items/mbid-f3c7e885-e459-43e5-865c-95c62a9781b9/mbid-f3c7e885-e459-43e5-865c-95c62a9781b9-19835076126.jpg
*   Trying 172.16.2.30...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 172.16.2.30 (172.16.2.30) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to ia803109.us.archive.org:443
> CONNECT ia803109.us.archive.org:443 HTTP/1.1
> Host: ia803109.us.archive.org:443
> User-Agent: curl/7.58.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection Established
< Proxy-Agent: IWSS
< Date: Sun, 08 Dec 2019 14:31:02 GMT
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* CONNECT phase completed!
* CONNECT phase completed!
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [80 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4773 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: OU=Domain Control Validated; CN=*.us.archive.org
*  start date: Dec 19 20:41:01 2016 GMT
*  expire date: Feb 21 22:56:17 2020 GMT
*  subjectAltName: host "ia803109.us.archive.org" matched cert's "*.us.archive.org"
*  issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2
*  SSL certificate verify ok.
} [5 bytes data]
> GET /33/items/mbid-f3c7e885-e459-43e5-865c-95c62a9781b9/mbid-f3c7e885-e459-43e5-865c-95c62a9781b9-19835076126.jpg HTTP/1.1
> Host: ia803109.us.archive.org
> User-Agent: curl/7.58.0
> Accept: */*
> 
{ [5 bytes data]
< HTTP/1.1 200 OK
< Server: nginx/1.14.0 (Ubuntu)
< Date: Sun, 08 Dec 2019 14:30:51 GMT
< Content-Type: image/jpeg
< Content-Length: 2126299
< Last-Modified: Wed, 09 May 2018 15:27:53 GMT
< Connection: keep-alive
< ETag: "5af31379-2071db"
< Expires: Sun, 08 Dec 2019 20:30:51 GMT
< Cache-Control: max-age=21600
< Access-Control-Allow-Origin: *
< Accept-Ranges: bytes
< 
{ [16027 bytes data]
100 2076k  100 2076k    0     0   375k      0  0:00:05  0:00:05 --:--:--  457k
* Connection #0 to host 172.16.2.30 left intact
slimshady@Slim-Shady:~/Downloads$ 
```
Hmm; intriguing! It really does seem like this must have to do with the proxies. A few things come to mind:

- Any chance you could try in some sort of temporary configuration that does not require a proxy (e.g., from a cafe's WiFi, or via a VPN)?
- The only thing I can think of that could be causing this is that we have a somewhat complex use of [Requests](https://requests.readthedocs.io/en/master/) for this plugin. Namely, we create an explicit session _and_ explicitly prepare the request. The [similar code in requests' own version of Session.request](https://github.com/psf/requests/blob/afb508f6a77c0f31b7e318c34166662cbf0a5844/requests/sessions.py#L516-L522) calls `self.merge_environment_settings` to ensure that the proxy stuff gets set up from the environment. Maybe we need to be doing that too, or find some way to avoid the manual steps we use.
Yes I did try with an internet source without proxy, it worked. Here is what I have. I also thought it might be some specific setting of my own laptop so I tried to run it on my other pc too. It is the same error so I am thinking it has nothing to do with a specific system.

```sh
slimshady@Slim-Shady:~$ beet -vv fetchart Deadpool\ 2
user configuration: /home/slimshady/.config/beets/config.yaml
data directory: /home/slimshady/.config/beets
plugin paths: /home/slimshady/Apps/beets/plugins
Sending event: pluginload
library database: /home/slimshady/Apps/beets/musiclibrary.db
library directory: /home/slimshady/Music
Sending event: library_opened
fetchart: trying source filesystem for album Various Artists - Deadpool 2: Original Motion Picture Soundtrack
fetchart: trying source itunes for album Various Artists - Deadpool 2: Original Motion Picture Soundtrack
fetchart: getting URL: https://itunes.apple.com/search?term=Various+Artists+Deadpool+2%3A+Original+Motion+Picture+Soundtrack&entity=album&media=music&limit=200
fetchart: downloading image: https://is4-ssl.mzstatic.com/image/thumb/Music118/v4/1f/92/3c/1f923c51-9785-39dd-42ac-d8e92e37fe7c/source/1200x1200bb.jpg
fetchart: downloaded art to: /tmp/tmpoz64bk2_.jpg
fetchart: using remote image /tmp/tmpoz64bk2_.jpg
Sending event: art_set
embedart: Embedding album art into Various Artists - Deadpool 2: Original Motion Picture Soundtrack
embedart: embedding /home/slimshady/Music/Compilations/Deadpool 2_ Original Motion Picture Soundtrack/cover.jpg
Sending event: write
Sending event: after_write
Sending event: database_change
fetchart: Various Artists - Deadpool 2: Original Motion Picture Soundtrack: found album art
Sending event: cli_exit
```
What is the benefit of doing what you do over what has been done in their own version? Sorry I am unaware and am not good at python. I only use this package.
Shockingly, the whole reason for doing things in this non-standard way is so that we can easily log the requested URL before actually connecting to the server:
https://github.com/beetbox/beets/blob/2e24887f539302f44e37dcf0119a8cb1f26a3150/beetsplug/fetchart.py#L170

This is really nice for transparency, but it requires this weird hack to get the fully-formed URL given the request parameters.

Anyway, it really seems like this is the culprit that's ignoring proxies! Seems like we need to do one of two things:

- Add in the `merge_environment_settings` call I mentioned above.
- Find an easier way to do our logging that avoids the need.

Actually, the latter thing might be easy: I think we can do `req.prepare()` even without creating an explicit `Session`. Now accepting PRs! :smiley:

Resolving patch:
diff --git a/beetsplug/fetchart.py b/beetsplug/fetchart.py
--- a/beetsplug/fetchart.py
+++ b/beetsplug/fetchart.py
@@ -164,9 +164,14 @@ def _logged_get(log, *args, **kwargs):
         message = 'getting URL'
 
     req = requests.Request('GET', *args, **req_kwargs)
+
     with requests.Session() as s:
         s.headers = {'User-Agent': 'beets'}
         prepped = s.prepare_request(req)
+        settings = s.merge_environment_settings(
+        	prepped.url, {}, None, None, None
+        )
+        send_kwargs.update(settings)
         log.debug('{}: {}', message, prepped.url)
         return s.send(prepped, **send_kwargs)
 


Test patch: