Repository: iterative/dvc

Issue:
'dvc gc -c' command not removing unused files if remote path is a subfolder of a S3 bucket
DVC Version: 0.54.1
Platform: Windows (exe)

If I have the remote path set to a subfolder of a S3 bucket and I execute `dvc gc -c`, none unused file will be removed. They are deleted in local cache but no in remote. However, if I execute the same command having set the root of a S3 bucket as remote path, unused files are removed fine in local and remote. I guess this is a bug because I don't see any explicit limitation about it on documentation.

Let me know if you need more info.


Discussion:
Hi @FranCarrascosaMS !

I'm not able to reproduce this. Here is my script:
```
#!/bin/bash                                   
                                              
set -e                                        
set -x                                        
                                              
rm -rf myrepo                                 
mkdir myrepo                                  
cd myrepo                                     
                                              
git init                                      
dvc init                                      
                                              
url="s3://dvc-test/$(uuidgen)"                
                                              
dvc remote add mys3 -d $url                   
                                              
echo foo > foo                                
dvc add foo                                   
                                              
mkdir data_dir                                
echo data > data_dir/data                     
                                              
mkdir data_dir/data_sub_dir                   
echo data_sub > data_dir/data_sub_dir/data_sub
                                              
dvc add data_dir                              
                                              
dvc push                                      
                                              
dvc remove -p foo.dvc                         
dvc remove -p data_dir.dvc                    
                                              
aws s3 ls $url --recursive                    
                                              
dvc gc -c                                     
                                              
aws s3 ls $url --recursive                    
```

which shows files being deleted, as expected.
ah, wait, I guess you meant a deeper level of subdir. Just a sec...
Nope, works the same with `url="s3://dvc-test/$(uuidgen)/$(uuidgen)"`. It would be great if we could find a minimal working reproducer for this issue, so far I'm not able to reproduce.
Ok, I think the issue happens when the folder is created manually through S3 console web.

Output of `aws s3 ls $url --recursive` after pushing a file when subfolder doesn't exist on S3:
![image](https://user-images.githubusercontent.com/32394038/62965085-94f8f100-be04-11e9-8d44-b8b9114a4f0d.png)

Output of `aws s3 ls $url --recursive` after pushing a file when subfolder was created manually:
![image](https://user-images.githubusercontent.com/32394038/62965091-99250e80-be04-11e9-9f89-17e80e4fcc7d.png)

You can close the issue if you want, your answer is enough for me :D Thank you!

@FranCarrascosaMS Got it. Hm, that is very interesting. I always thought that s3 doesn't really have `folders` as a separate entity, but looks like I was wrong. Here is a script that fails on `gc` with an error:
```
#!/bin/bash                                           
                                                      
set -e                                                
set -x                                                
                                                      
rm -rf myrepo                                         
mkdir myrepo                                          
cd myrepo                                             
                                                      
git init                                              
dvc init                                              
                                                      
prefix="$(uuidgen)"                                   
url="s3://dvc-test/$prefix"                           
                                                      
aws s3api put-object --bucket dvc-test --key $prefix/ 
                                                      
dvc remote add mys3 -d $url                           
                                                      
echo foo > foo                                        
dvc add foo                                           
                                                      
mkdir data_dir                                        
echo data > data_dir/data                             
                                                      
mkdir data_dir/data_sub_dir                           
echo data_sub > data_dir/data_sub_dir/data_sub        
                                                      
dvc add data_dir                                      
                                                      
dvc push                                              
                                                      
dvc remove -p foo.dvc                                 
dvc remove -p data_dir.dvc                            
                                                      
aws s3 ls $url --recursive                            
                                                      
dvc gc -c                                             
                                                      
aws s3 ls $url --recursive                            
```

Error:

```
ERROR: unexpected error - sequence item 0: expected str instance, NoneType found                                                         
------------------------------------------------------------                                                                             
Traceback (most recent call last):                                                                                                       
  File "/home/efiop/.pyenv/versions/3.7.0/envs/3.7.0-dvc/lib/python3.7/site-packages/dvc/main.py", line 40, in main                      
    ret = cmd.run_cmd()                                                                                                                  
  File "/home/efiop/.pyenv/versions/3.7.0/envs/3.7.0-dvc/lib/python3.7/site-packages/dvc/command/base.py", line 63, in run_cmd           
    return self.run()                                                                                                                    
  File "/home/efiop/.pyenv/versions/3.7.0/envs/3.7.0-dvc/lib/python3.7/site-packages/dvc/command/gc.py", line 47, in run                 
    repos=self.args.repos,                                                                                                               
  File "/home/efiop/.pyenv/versions/3.7.0/envs/3.7.0-dvc/lib/python3.7/site-packages/dvc/repo/gc.py", line 113, in gc                    
    _do_gc("remote", self.cloud.get_remote(remote, "gc -c").gc, clist)                                                                   
  File "/home/efiop/.pyenv/versions/3.7.0/envs/3.7.0-dvc/lib/python3.7/site-packages/dvc/repo/gc.py", line 54, in _do_gc                 
    removed = func(clist)                                                                                                                
  File "/home/efiop/.pyenv/versions/3.7.0/envs/3.7.0-dvc/lib/python3.7/site-packages/dvc/remote/base.py", line 542, in gc                
    for checksum in self.all():                                                                                                          
  File "/home/efiop/.pyenv/versions/3.7.0/envs/3.7.0-dvc/lib/python3.7/site-packages/dvc/remote/base.py", line 529, in <genexpr>         
    self.path_to_checksum(path) for path in self.list_cache_paths()                                                                      
  File "/home/efiop/.pyenv/versions/3.7.0/envs/3.7.0-dvc/lib/python3.7/site-packages/dvc/remote/base.py", line 515, in path_to_checksum  
    return "".join(self.path_cls(path).parts[-2:])                                                                                       
TypeError: sequence item 0: expected str instance, NoneType found                                                                        
```

So it is indeed a bug on our side. Thanks for reporting! We'll take a look.

Resolving patch:
diff --git a/dvc/remote/base.py b/dvc/remote/base.py
--- a/dvc/remote/base.py
+++ b/dvc/remote/base.py
@@ -516,7 +516,12 @@ def exists(self, path_info):
         raise NotImplementedError
 
     def path_to_checksum(self, path):
-        return "".join(self.path_cls(path).parts[-2:])
+        parts = self.path_cls(path).parts[-2:]
+
+        if not (len(parts) == 2 and parts[0] and len(parts[0]) == 2):
+            raise ValueError("Bad cache file path")
+
+        return "".join(parts)
 
     def checksum_to_path_info(self, checksum):
         return self.path_info / checksum[0:2] / checksum[2:]
@@ -529,9 +534,12 @@ def all(self):
         # is 32 bytes, so ~3200Mb list) and we don't really need all of it at
         # the same time, so it makes sense to use a generator to gradually
         # iterate over it, without keeping all of it in memory.
-        return (
-            self.path_to_checksum(path) for path in self.list_cache_paths()
-        )
+        for path in self.list_cache_paths():
+            try:
+                yield self.path_to_checksum(path)
+            except ValueError:
+                # We ignore all the non-cache looking files
+                pass
 
     def gc(self, cinfos):
         used = self.extract_used_local_checksums(cinfos)


Test patch: