Repository: rucio/rucio

Issue:
Client: Check URL escaping code for handling of /
If we want `/` to be a valid character in the LFN, URL escaping in the client suddenly becomes more relevant.

For example, on the CMS test instance, I noticed the following GET request in the logs:

```
GET /dids/user.bbockelm//store/mc/SAM/GenericTTbar/AODSIM/CMSSW_9_2_6_91X_mcRun1_realistic_v2-v1/00000/AE237916-5D76-E711-A48C-FA163EEEBFED.root/meta HTTP/1.1"
```

We probably wanted to issue this request:

```
GET /dids/user.bbockelm/%2Fstore%2Fmc%2FSAM%2FGenericTTbar%2FAODSIM%2FCMSSW_9_2_6_91X_mcRun1_realistic_v2-v1%2F00000%2FAE237916-5D76-E711-A48C-FA163EEEBFED.root/meta HTTP/1.1"
```


Discussion:
For example, I think we'll need a lot of patches along this line:
```
--- a/lib/rucio/client/didclient.py
+++ b/lib/rucio/client/didclient.py
@@ -15,6 +15,7 @@
   - Martin Barisits, <martin.barisits@cern.ch>, 2014-2015
 '''
 
+from urllib import quote_plus
 from json import dumps
 from requests.status_codes import codes
 
@@ -375,7 +376,7 @@ class DIDClient(BaseClient):
         :param scope: The scope name.
         :param name: The data identifier name.
         """
-        path = '/'.join([self.DIDS_BASEURL, scope, name, 'meta'])
+        path = '/'.join([self.DIDS_BASEURL, quote_plus(scope), quote_plus(name), 'meta'])
         url = build_url(choice(self.list_hosts), path=path)
         r = self._send_request(url, type='GET')
         if r.status_code == codes.ok:
```
Notes:

- Not sure if we want to quote the scope or keep that relatively restricted.
- There's a corresponding Apache-side change to allow encoded slashes in the URL.
Ok, some progress here.

Once the client properly escapes `/` to `%2F` and Apache doesn't change the URL, the next step is `mod_wsgi`: this again escapes and automatically "cleans up" `//` characters.  So,

```
/dids/user.bbockelm//foo/bar/baz/meta
```

will, with some regex hackery, invoke the right function with scope `user.bbockelm` and DID `foo/bar/baz` (note missing prefix `/`).

Since Rucio already has a pre-load hook, one can correct the path and make WSGI "do the right thing".  Example code for the hook:

```
    request_uri = ctx.env.get('REQUEST_URI', '')
    script_name = ctx.env.get('SCRIPT_NAME', '')
    if request_uri.startswith(script_name):
        path_part = request_uri.split("?")[0]
        path_part = normpath(path_part[len(script_name):])
        while path_part.startswith("//"):
            path_part = path_part[1:]
        ctx.path = unquote(path_part)
```

This causes the right handler to be invoked with scope `user.bbockelm` and DID `/foo/bar/baz`.

It works with the caveat that one has to tweak the regex for the WSGI application.  Instead of:

```
    '/(.*)/(.*)/meta', 'Meta',
```

one needs:

```
    '/([^/]*)/(.*)/meta', 'Meta',
```

(see `URLS` global in `lib/rucio/web/rest/did.py`)

So, that's a modest project to do minor fixups to all APIs that include a DID (with an ugly hack in the load hook).  Definitely an option, but not a particularly appealing one.

The second option is to completely rework the APIs to allow them to accept either query-style or path-style information.  That is, have:

```
/dids/meta?scope=user.bbockelm&did=/foo/bar/baz
```

equivalent to

```
/dids/user.bbockelm//foo/bar/baz/meta
```

One can keep backward compatibility with existing clients.  It's more work, but seems cleaner to me.

Thoughts?
one third alternative is to have:
```'/(.*)/meta', 'Meta',```

and a function splitting the obtained string in scope, name -- assuming that scope doesn't have a /. we can stay backward compatible with this change.

going for:
```/dids/meta?scope=user.bbockelm&did=/foo/bar/baz```

will need to introduce an api version (api version variable in the http header or in the url /v2/dids/meta?..). 
@vingar - no, unfortunately that won't work due to the substitution of the the `//` into `/`.  The resulting DID is `foo/bar/baz` instead of `/foo/bar/baz`.  You end up having to do the same loader hook trick I highlighted above (which again - it does indeed work without client changes, just I find it ugly).

Now, having an API version is a solid idea regardless - but it's not clear why `/dids/meta?scope=user.bbockelm&did=/foo/bar/baz` would require one?  That's not a valid request currently.

Or - do you mean that we'd want to introduce an API version so the client can more easily auto-detect the version of the server (and hence the API it wants to use)?
hi, I need to go in the details of the code to know what is doable. Unfortunately, this discussion doesn't concern only the metadata part. The same logic has been applied to several resources (dids, replicas, rules, etc) and http operations (put/get/post/del).
In the meantime I'm trying to figure out if we can configure mod_wsgi not do the slash escape.
@bbockelm This looks like it: http://httpd.apache.org/docs/2.2/mod/core.html#allowencodedslashes
Another option, which should keep backward compatibility with the clients:

with special-lookahead in regexp, we can extract the delimiter from the regexp. Example:
```
>>> re.match('/([^/]*)(?=/)(.*)/meta', '/user.bbockelm/foo/bar/baz/meta').groups()
('user.bbockelm', '/foo/bar/baz')
```
one possibility then is to make certain regexp policy configurable. 

Resolving patch:
diff --git a/lib/rucio/core/did.py b/lib/rucio/core/did.py
--- a/lib/rucio/core/did.py
+++ b/lib/rucio/core/did.py
@@ -1157,6 +1157,16 @@ def set_metadata(scope, name, key, value, type=None, did=None,
             session.query(models.DataIdentifier).filter_by(scope=scope, name=name).update({'expired_at': expired_at}, synchronize_session='fetch')
         except TypeError as error:
             raise exception.InvalidValueForKey(error)
+    elif key in ['guid', 'events']:
+        rowcount = session.query(models.DataIdentifier).filter_by(scope=scope, name=name, did_type=DIDType.FILE).update({key: value}, synchronize_session=False)
+        if not rowcount:
+            raise exception.UnsupportedOperation('%(key)s for %(scope)s:%(name)s cannot be updated' % locals())
+        session.query(models.DataIdentifierAssociation).filter_by(child_scope=scope, child_name=name, child_type=DIDType.FILE).update({key: value}, synchronize_session=False)
+        if key == 'events':
+            for parent_scope, parent_name in session.query(models.DataIdentifierAssociation.scope, models.DataIdentifierAssociation.name).filter_by(child_scope=scope, child_name=name):
+                events = session.query(func.sum(models.DataIdentifierAssociation.events)).filter_by(scope=parent_scope, name=parent_name).one()
+                session.query(models.DataIdentifier).filter_by(scope=parent_scope, name=parent_name).update({'events': events}, synchronize_session=False)
+
     elif key == 'adler32':
         rowcount = session.query(models.DataIdentifier).filter_by(scope=scope, name=name, did_type=DIDType.FILE).update({key: value}, synchronize_session=False)
         if not rowcount:
diff --git a/lib/rucio/daemons/atropos.py b/lib/rucio/daemons/atropos.py
--- a/lib/rucio/daemons/atropos.py
+++ b/lib/rucio/daemons/atropos.py
@@ -22,7 +22,6 @@
 from rucio.db.sqla.constants import LifetimeExceptionsState
 from rucio.common.config import config_get
 from rucio.common.exception import RuleNotFound
-import rucio.common.policy
 from rucio.core import heartbeat
 import rucio.core.lifetime_exception
 from rucio.core.lock import get_dataset_locks
@@ -87,47 +86,50 @@ def atropos(thread, bulk, date_check, dry_run=True, grace_period=86400, once=Tru
 
                     if (rule_idx % 1000) == 0:
                         logging.info(prepend_str + '%s/%s rules processed' % (rule_idx, len(rules)))
-                    # We compute the expended eol_at
+                    # We compute the expected eol_at
                     rses = parse_expression(rule.rse_expression)
                     eol_at = rucio.core.lifetime_exception.define_eol(rule.scope, rule.name, rses)
+                    if eol_at != rule.eol_at:
+                        logging.warning(prepend_str + 'The computed eol %s differs from the one recorded %s for rule %s on %s at %s' % (eol_at, rule.eol_at, rule.id,
+                                                                                                                                        did, rule.rse_expression))
+                        try:
+                            update_rule(rule.id, options={'eol_at': eol_at})
+                        except RuleNotFound:
+                            logging.warning(prepend_str + 'Cannot find rule %s on DID %s' % (rule.id, did))
 
                     # Check the exceptions
                     if did in lifetime_exceptions:
-                        if rule.eol_at > lifetime_exceptions[did]:
-                            logging.info(prepend_str + 'Rule %s on DID %s on %s expired. Extension requested till %s' % (rule.id, did, rule.rse_expression,
-                                                                                                                         lifetime_exceptions[did]))
+                        if eol_at > lifetime_exceptions[did]:
+                            logging.info(prepend_str + 'Rule %s on DID %s on %s has longer expiration date than the one requested : %s' % (rule.id, did, rule.rse_expression,
+                                                                                                                                           lifetime_exceptions[did]))
                         else:
                             # If eol_at < requested extension, update eol_at
                             logging.info(prepend_str + 'Updating rule %s on DID %s on %s according to the exception till %s' % (rule.id, did, rule.rse_expression,
                                                                                                                                 lifetime_exceptions[did]))
+                            eol_at = lifetime_exceptions[did]
                             try:
                                 update_rule(rule.id, options={'eol_at': lifetime_exceptions[did]})
                             except RuleNotFound:
                                 logging.warning(prepend_str + 'Cannot find rule %s on DID %s' % (rule.id, did))
-                    elif eol_at != rule.eol_at:
-                        logging.warning(prepend_str + 'The computed eol %s differs from the one recorded %s for rule %s on %s at %s' % (eol_at, rule.eol_at, rule.id,
-                                                                                                                                        did, rule.rse_expression))
-                        try:
-                            update_rule(rule.id, options={'eol_at': eol_at})
-                        except RuleNotFound:
-                            logging.warning(prepend_str + 'Cannot find rule %s on DID %s' % (rule.id, did))
 
-                    no_locks = True
-                    for lock in get_dataset_locks(rule.scope, rule.name):
-                        if lock['rule_id'] == rule[4]:
-                            no_locks = False
-                            if lock['rse'] not in summary:
-                                summary[lock['rse']] = {}
-                            if did not in summary[lock['rse']]:
-                                summary[lock['rse']][did] = {'length': lock['length'] or 0, 'bytes': lock['bytes'] or 0}
-                    if no_locks:
-                        logging.warning(prepend_str + 'Cannot find a lock for rule %s on DID %s' % (rule.id, did))
-                    if not dry_run:
-                        logging.info(prepend_str + 'Setting %s seconds lifetime for rule %s' % (grace_period, rule.id))
-                        try:
-                            update_rule(rule.id, options={'lifetime': grace_period})
-                        except RuleNotFound:
-                            logging.warning(prepend_str + 'Cannot find rule %s on DID %s' % (rule.id, did))
+                    # Now check that the new eol_at is expired
+                    if eol_at < date_check:
+                        no_locks = True
+                        for lock in get_dataset_locks(rule.scope, rule.name):
+                            if lock['rule_id'] == rule[4]:
+                                no_locks = False
+                                if lock['rse'] not in summary:
+                                    summary[lock['rse']] = {}
+                                if did not in summary[lock['rse']]:
+                                    summary[lock['rse']][did] = {'length': lock['length'] or 0, 'bytes': lock['bytes'] or 0}
+                        if no_locks:
+                            logging.warning(prepend_str + 'Cannot find a lock for rule %s on DID %s' % (rule.id, did))
+                        if not dry_run:
+                            logging.info(prepend_str + 'Setting %s seconds lifetime for rule %s' % (grace_period, rule.id))
+                            try:
+                                update_rule(rule.id, options={'lifetime': grace_period})
+                            except RuleNotFound:
+                                logging.warning(prepend_str + 'Cannot find rule %s on DID %s' % (rule.id, did))
             except Exception:
                 exc_type, exc_value, exc_traceback = exc_info()
                 logging.critical(''.join(format_exception(exc_type, exc_value, exc_traceback)).strip())


Test patch: