Repository: rucio/rucio

Issue:
posix protocol modifications
Motivation
----------

in ATLAS StoRM sites can can read directly from the file system (probably write too) but the path resolution is done via storm server. 

Some sites setting up cephFS would like to read directly from the FS but there is no possibility at the moment.  

Modification
------------

Modify posix protocol to allow direct access to the fs. At the moment it copies the files but the best thing would be to have a symlink like storm sites but without needing to get the information from a storage server.

this would simplify the setup at sites removing the need of extra hardware to handle the read and possibly write to the file system.



Discussion:
This one needs a symlink subclass on the posix protocol, and overwrite the get() with a symlink variant, and a overwrite the exists() to check that not the symlink but the file followed by a symlink actually exists.
I would like to work on this, but I feel like I may need some pointers on where to get started.
Hi, 
  I've started work on this topic here: 
 https://github.com/snafus/rucio/tree/patch-5319-PosixWithSymlinkProtocolImpl 
 The current version appears to demonstrate the required functionality, and I hope to make a PR shortly. 
 Any comments are happily received at this point.
 https://github.com/snafus/rucio/tree/patch-5319-PosixWithSymlinkProtocolImpl 
 
 I did not overwrite the exists() method, as this gets the path from PFN of the 'remote' file; but let me know if I missed something. 
 Thanks,
 James
Hi, I've updated the pull request with a couple of fixes for logging output formatting. 

In terms of testing; I've tried the following:
```
rucio-admin rse add POS1
rucio-admin rse add-protocol --hostname "" --scheme file --prefix /tmp/pos1  --impl rucio.rse.protocols.posix.Symlink --domain-json '{"wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy": 1}, "lan": {"read": 1, "write": 1, "delete": 1}}' POS1
mkdir /tmp/pos1
rucio-admin rse info POS1

dd if=/dev/urandom of=fpos1 bs=10M count=1
dd if=/dev/urandom of=fpos2 bs=10M count=1

rucio upload --rse POS1 --scope test fpos1
rucio upload --rse POS1 --scope test fpos2
rucio add-dataset test:posdataset1
rucio attach test:posdataset1 test:fpos1 test:fpos2

rucio list-file-replicas test:posdataset1

rucio download test:posdataset1
ls -ltrh posdataset1
```

With outputs of 
```
rucio list-file-replicas --protocol file test:posdataset1
+---------+--------+------------+-----------+-----------------------------------------+
| SCOPE   | NAME   | FILESIZE   | ADLER32   | RSE: REPLICA                            |
|---------+--------+------------+-----------+-----------------------------------------|
| test    | fpos1  | 10.486 MB  | fbf84b82  | POS1: file:///tmp/pos1/test/16/f4/fpos1 |
| test    | fpos2  | 10.486 MB  | 211742a4  | POS1: file:///tmp/pos1/test/cb/41/fpos2 |
+---------+--------+------------+-----------+-----------------------------------------+

rucio download test:posdataset1
ls -ltrh posdataset1/
total 0
lrwxrwxrwx 1 root root 26 May 18 05:09 fpos1 -> /tmp/pos1/test/16/f4/fpos1
lrwxrwxrwx 1 root root 26 May 18 05:09 fpos2 -> /tmp/pos1/test/cb/41/fpos2
```

James

Hi James, sorry, completely missed this. Let me have a closer look.
Hi @mlassnig ; I've added some slightly improved exception handling, and a basic unit test (although I think the get methods are not yet available?).
I hope this is a suitable version to be considered. Thanks, James

Resolving patch:
diff --git a/lib/rucio/rse/protocols/posix.py b/lib/rucio/rse/protocols/posix.py
--- a/lib/rucio/rse/protocols/posix.py
+++ b/lib/rucio/rse/protocols/posix.py
@@ -13,6 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import logging
 import os
 import os.path
 import shutil
@@ -210,3 +211,41 @@ def stat(self, pfn):
         """
         path = self.pfn2path(pfn)
         return {'filesize': os.stat(path)[os.path.stat.ST_SIZE], 'adler32': adler32(path)}
+
+
+class Symlink(Default):
+    """ Implementing access to RSEs using the local filesystem, creating a symlink on a get """
+
+    def get(self, pfn, dest, transfer_timeout=None):
+        """ Provides access to files stored inside connected the RSE.
+            A download/get will create a symlink on the local file system pointing to the
+            underlying file. Other operations act directly on the remote file.
+            :param pfn: Physical file name of requested file
+            :param dest: Name and path of the files when stored at the client
+            :param transfer_timeout Transfer timeout (in seconds) - dummy
+            :raises DestinationNotAccessible: if the destination storage was not accessible.
+            :raises ServiceUnavailable: if some generic error occured in the library.
+            :raises SourceNotFound: if the source file was not found on the referred storage.
+         """
+        path = self.pfn2path(pfn)
+        os.symlink(path, dest)
+        self.logger(logging.DEBUG,
+                    'Symlink {} created for {} from {}'
+                    .format(dest, path, pfn))
+        if not os.lstat(dest):
+            # problem in creating the symlink
+            self.logger(logging.ERROR, 'Symlink {} could not be created'.format(dest))
+            raise exception.DestinationNotAccessible()
+        if not os.path.exists(dest):
+            # could not find the file following the symlink
+            self.logger(logging.ERROR, 'Symlink {} appears to be a broken link to {}'
+                        .format(dest, path))
+            if os.lstat(dest) and os.path.islink(dest):
+                os.unlink(dest)
+            raise exception.SourceNotFound()
+
+    def pfn2path(self, pfn):
+        # obtain path and sanitise from multiple slashes, etc
+        path = os.path.normpath(super().pfn2path(pfn))
+        self.logger(logging.DEBUG, 'Extracted path: {} from: {}'.format(path, pfn))
+        return path


Test patch:
diff --git a/lib/rucio/tests/test_rse_protocol_posix.py b/lib/rucio/tests/test_rse_protocol_posix.py
--- a/lib/rucio/tests/test_rse_protocol_posix.py
+++ b/lib/rucio/tests/test_rse_protocol_posix.py
@@ -233,3 +233,186 @@ def test_change_scope_mgr_ok_single_lfn(self):
     def test_change_scope_mgr_ok_single_pfn(self):
         """POSIX (RSE/PROTOCOLS): Change the scope of a single file on storage using PFN (Success)"""
         self.mtc.test_change_scope_mgr_ok_single_pfn()
+
+
+@skip_rse_tests_with_accounts
+@pytest.mark.noparallel(reason='creates and removes a test directory with a fixed name')
+class TestRsePOSIXSymlink(unittest.TestCase):
+    """
+    Test the posix protocol using Symlink implementation
+    """
+
+    tmpdir = None
+    user = None
+
+    @classmethod
+    def setUpClass(cls):
+        """POSIX-SYMLINK (RSE/PROTOCOLS): Creating necessary directories and files """
+        # Creating local files
+        cls.tmpdir = tempfile.mkdtemp()
+        cls.user = uuid()
+
+        with open("%s/data.raw" % cls.tmpdir, "wb") as out:
+            out.seek((1024 * 1024) - 1)  # 1 MB
+            out.write(b'\0')
+        for f in MgrTestCases.files_local:
+            shutil.copy('%s/data.raw' % cls.tmpdir, '%s/%s' % (cls.tmpdir, f))
+
+        data = load_test_conf_file('rse_repository.json')
+        prefix = data['MOCK-POSIX-SYMLINK']['protocols']['supported']['file']['prefix']
+        try:
+            os.mkdir(prefix)
+        except Exception as e:
+            print(e)
+        os.system('dd if=/dev/urandom of=%s/data.raw bs=1024 count=1024' % prefix)
+        cls.static_file = '%s/data.raw' % prefix
+        for f in MgrTestCases.files_remote:
+            protocol = mgr.create_protocol(mgr.get_rse_info('MOCK-POSIX-SYMLINK'), 'write')
+            pfn = list(mgr.lfns2pfns(mgr.get_rse_info('MOCK-POSIX-SYMLINK'),
+                                     {'name': f, 'scope': 'user.%s' % cls.user}).values())[0]
+            path = protocol.pfn2path(pfn)
+            dirs = os.path.dirname(path)
+            if not os.path.exists(dirs):
+                os.makedirs(dirs)
+            shutil.copy('%s/data.raw' % prefix, path)
+
+    @classmethod
+    def tearDownClass(cls):
+        """POSIX (RSE/PROTOCOLS): Removing created directories and files """
+        data = load_test_conf_file('rse_repository.json')
+        prefix = data['MOCK-POSIX-SYMLINK']['protocols']['supported']['file']['prefix']
+        shutil.rmtree(prefix)
+        shutil.rmtree(cls.tmpdir)
+
+    def setUp(self):
+        """POSIX (RSE/PROTOCOLS): Creating Mgr-instance """
+        self.tmpdir = self.__class__.tmpdir
+        self.rse_id = 'MOCK-POSIX-SYMLINK'
+        self.mtc = MgrTestCases(self.tmpdir, 'MOCK-POSIX-SYMLINK', self.__class__.user, self.__class__.static_file)
+
+    # # Mgr-Tests: GET
+    # Get tests will fail as rsemanager.download has been removed
+
+    # Mgr-Tests: PUT
+    def test_put_mgr_ok_multi(self):
+        """POSIX (RSE/PROTOCOLS): Put multiple files to storage providing LFNs and PFNs (Success)"""
+        self.mtc.test_put_mgr_ok_multi()
+
+    def test_put_mgr_ok_single(self):
+        """POSIX (RSE/PROTOCOLS): Put a single file to storage (Success)"""
+        self.mtc.test_put_mgr_ok_single()
+
+    def test_put_mgr_SourceNotFound_multi(self):
+        """POSIX (RSE/PROTOCOLS): Put multiple files to storage (SourceNotFound)"""
+        with pytest.raises(exception.SourceNotFound):
+            self.mtc.test_put_mgr_SourceNotFound_multi()
+
+    def test_put_mgr_SourceNotFound_single(self):
+        """POSIX (RSE/PROTOCOLS): Put a single file to storage (SourceNotFound)"""
+        with pytest.raises(exception.SourceNotFound):
+            self.mtc.test_put_mgr_SourceNotFound_single()
+
+    def test_put_mgr_FileReplicaAlreadyExists_multi(self):
+        """POSIX (RSE/PROTOCOLS): Put multiple files to storage (FileReplicaAlreadyExists)"""
+        with pytest.raises(exception.FileReplicaAlreadyExists):
+            self.mtc.test_put_mgr_FileReplicaAlreadyExists_multi()
+
+    def test_put_mgr_FileReplicaAlreadyExists_single(self):
+        """POSIX (RSE/PROTOCOLS): Put a single file to storage (FileReplicaAlreadyExists)"""
+        with pytest.raises(exception.FileReplicaAlreadyExists):
+            self.mtc.test_put_mgr_FileReplicaAlreadyExists_single()
+
+    # MGR-Tests: DELETE
+    def test_delete_mgr_ok_multi(self):
+        """POSIX (RSE/PROTOCOLS): Delete multiple files from storage (Success)"""
+        self.mtc.test_delete_mgr_ok_multi()
+
+    def test_delete_mgr_ok_single(self):
+        """POSIX (RSE/PROTOCOLS): Delete a single file from storage (Success)"""
+        self.mtc.test_delete_mgr_ok_single()
+
+    def test_delete_mgr_SourceNotFound_multi(self):
+        """POSIX (RSE/PROTOCOLS): Delete multiple files from storage (SourceNotFound)"""
+        with pytest.raises(exception.SourceNotFound):
+            self.mtc.test_delete_mgr_SourceNotFound_multi()
+
+    def test_delete_mgr_SourceNotFound_single(self):
+        """POSIX (RSE/PROTOCOLS): Delete a single file from storage (SourceNotFound)"""
+        with pytest.raises(exception.SourceNotFound):
+            self.mtc.test_delete_mgr_SourceNotFound_single()
+
+    # MGR-Tests: EXISTS
+    def test_exists_mgr_ok_multi(self):
+        """POSIX (RSE/PROTOCOLS): Check multiple files on storage (Success)"""
+        self.mtc.test_exists_mgr_ok_multi()
+
+    def test_exists_mgr_ok_single_lfn(self):
+        """POSIX (RSE/PROTOCOLS): Check a single file on storage using LFN (Success)"""
+        self.mtc.test_exists_mgr_ok_single_lfn()
+
+    def test_exists_mgr_ok_single_pfn(self):
+        """POSIX (RSE/PROTOCOLS): Check a single file on storage using PFN (Success)"""
+        self.mtc.test_exists_mgr_ok_single_pfn()
+
+    def test_exists_mgr_false_multi(self):
+        """POSIX (RSE/PROTOCOLS): Check multiple files on storage (Fail)"""
+        self.mtc.test_exists_mgr_false_multi()
+
+    def test_exists_mgr_false_single(self):
+        """POSIX (RSE/PROTOCOLS): Check a single file on storage using LFN (Fail)"""
+        self.mtc.test_exists_mgr_false_single_lfn()
+
+    def test_exists_mgr_false_single_pfn(self):
+        """POSIX (RSE/PROTOCOLS): Check a single file on storage using PFN (Fail)"""
+        self.mtc.test_exists_mgr_false_single_pfn()
+
+    # MGR-Tests: RENAME
+    def test_rename_mgr_ok_multi(self):
+        """POSIX (RSE/PROTOCOLS): Rename multiple files on storage (Success)"""
+        self.mtc.test_rename_mgr_ok_multi()
+
+    def test_rename_mgr_ok_single_lfn(self):
+        """POSIX (RSE/PROTOCOLS): Rename a single file on storage using LFN (Success)"""
+        self.mtc.test_rename_mgr_ok_single_lfn()
+
+    def test_rename_mgr_ok_single_pfn(self):
+        """POSIX (RSE/PROTOCOLS): Rename a single file on storage using PFN (Success)"""
+        self.mtc.test_rename_mgr_ok_single_pfn()
+
+    def test_rename_mgr_FileReplicaAlreadyExists_multi(self):
+        """POSIX (RSE/PROTOCOLS): Rename multiple files on storage (FileReplicaAlreadyExists)"""
+        with pytest.raises(exception.FileReplicaAlreadyExists):
+            self.mtc.test_rename_mgr_FileReplicaAlreadyExists_multi()
+
+    def test_rename_mgr_FileReplicaAlreadyExists_single_lfn(self):
+        """POSIX (RSE/PROTOCOLS): Rename a single file on storage using LFN(FileReplicaAlreadyExists)"""
+        with pytest.raises(exception.FileReplicaAlreadyExists):
+            self.mtc.test_rename_mgr_FileReplicaAlreadyExists_single_lfn()
+
+    def test_rename_mgr_FileReplicaAlreadyExists_single_pfn(self):
+        """POSIX (RSE/PROTOCOLS): Rename a single file on storage using PFN (FileReplicaAlreadyExists)"""
+        with pytest.raises(exception.FileReplicaAlreadyExists):
+            self.mtc.test_rename_mgr_FileReplicaAlreadyExists_single_pfn()
+
+    def test_rename_mgr_SourceNotFound_multi(self):
+        """POSIX (RSE/PROTOCOLS): Rename multiple files on storage (SourceNotFound)"""
+        with pytest.raises(exception.SourceNotFound):
+            self.mtc.test_rename_mgr_SourceNotFound_multi()
+
+    def test_rename_mgr_SourceNotFound_single_lfn(self):
+        """POSIX (RSE/PROTOCOLS): Rename a single file on storage using LFN (SourceNotFound)"""
+        with pytest.raises(exception.SourceNotFound):
+            self.mtc.test_rename_mgr_SourceNotFound_single_lfn()
+
+    def test_rename_mgr_SourceNotFound_single_pfn(self):
+        """POSIX (RSE/PROTOCOLS): Rename a single file on storage using PFN (SourceNotFound)"""
+        with pytest.raises(exception.SourceNotFound):
+            self.mtc.test_rename_mgr_SourceNotFound_single_pfn()
+
+    def test_change_scope_mgr_ok_single_lfn(self):
+        """POSIX (RSE/PROTOCOLS): Change the scope of a single file on storage using LFN (Success)"""
+        self.mtc.test_change_scope_mgr_ok_single_lfn()
+
+    def test_change_scope_mgr_ok_single_pfn(self):
+        """POSIX (RSE/PROTOCOLS): Change the scope of a single file on storage using PFN (Success)"""
+        self.mtc.test_change_scope_mgr_ok_single_pfn()