Repository: iterative/dvc

Issue:
`dvc pull` with wrong S3 remote failed but user wasn't informed
System
----------

My `dvc` version is 0.77.3, I installed & upgraded `dvc` through `pip3` and I'm using Fedora 31.

Problem
-----------

I followed the 'Getting Started' guide and ran the following commands

```
$ dvc init
$ git add .dvc/* && git commit -m 'Initialized dvc'
$ dvc remote add -d s3remote https://s3.amazonaws.com/<bucket-name>
$ dvc add <some-file>
$ git add .gitignore <some-file>.dvc && git commit -m 'Added <some-file'
$ dvc push -r s3remote
```
The last command finished with `Everything is up to date.` which is weird since the S3 bucket was empty when checking it. I went on to delete `<some-file>` and run `dvc pull -r s3remote` which would restore `<some-file>` without throwing any errors. This was surprising since it did not get clear where `dvc` is storing my file backups and why my S3 bucket was still empty without `dvc` communicating any errors.

Only when I removed the `.dvc/cache` folder and ran `dvc pull -r s3remote` it would complain with the following:

```
ERROR: failed to download 'https://s3.amazonaws.com/<bucket-name>/31/69f7ce4ebb503afca037d35b7eb3a9' to '.dvc/cache/31/69f7ce4ebb503afca037d35b7eb3a9' - '301 Moved Permanently'

ERROR: failed to download 'https://s3.amazonaws.com/<bucket-name>/a3/04afb96060aad90176268345e10355' to '.dvc/cache/a3/04afb96060aad90176268345e10355' - '301 Moved Permanently'

ERROR: failed to pull data from the cloud - 2 files failed to download


Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
```

The fix
---------

On my end, it was an easy fix. After consulting the documentation in more detail, I realized that I set up my S3 remote incorrectly using the `https://s3.amazonaws.com/<bucket-name>` URL to the bucket whereas I should have used the `s3://<bucket-name>` URL. Hence, running

```
$ dvc remote modify s3remote url s3://<bucket-name>
```

did the job.

It still remains weird why I was never warned about the remote being incorrectly set up and why `dvc pull -r s3remote` worked without any problems even though there were no files in the S3 bucket (it seems that it restored them from local cache as a fall-back) but it should have given me a headsup!


Discussion:
Hi @markf94 !

Thanks for reporting this issue! I can reproduce:

```
#!/bin/bash

set -e
set -x

rm -rf myrepo
mkdir myrepo
cd myrepo
git init
dvc init
git commit -m 'Initialized dvc'
dvc remote add -d s3remote https://s3.amazonaws.com/dvc-temp/ruslan
echo foo > foo
dvc add foo
git add .gitignore foo.dvc && git commit -m 'Added some-file'
dvc push
```

Same with `dvc status -c`. So there is definitely a bug, looking into it.

For the record, the sequential `pull` didn't complain because it saw that it already has everything it needs in the local cache, so it didn't have to download anything.

Hey there, I picked this one up!

I found a likely culprit - the existence check for an HTTP/S remote is not exactly thorough.

Resolving patch:
diff --git a/dvc/remote/http.py b/dvc/remote/http.py
--- a/dvc/remote/http.py
+++ b/dvc/remote/http.py
@@ -101,7 +101,22 @@ def _request(self, method, url, **kwargs):
         kwargs.setdefault("timeout", self.REQUEST_TIMEOUT)
 
         try:
-            return self._session.request(method, url, **kwargs)
+            res = self._session.request(method, url, **kwargs)
+
+            redirect_no_location = (
+                kwargs["allow_redirects"]
+                and res.status_code in (301, 302)
+                and "location" not in res.headers
+            )
+
+            if redirect_no_location:
+                # AWS s3 doesn't like to add a location header to its redirects
+                # from https://s3.amazonaws.com/<bucket name>/* type URLs.
+                # This should be treated as an error
+                raise requests.exceptions.RequestException
+
+            return res
+
         except requests.exceptions.RequestException:
             raise DvcException("could not perform a {} request".format(method))
 


Test patch: