Repository: rucio/rucio

Issue:
rucio upload CLI with parent dataset doesn't create rules
I may be misunderstanding the arguments to `rucio upload` here, but either way this doesn't seem like desirable behaviour.

From https://github.com/rucio/rucio/blob/master/bin/rucio#L1002, we have been operating under the belief we could upload a file and attach it to an existing dataset with `rucio upload <dataset-did> <file/folder>`. Although this seems to work (i.e. generates a replica), it doesn't seem to generate a replication rule for that replica:

```bash
[user@903de6bd49ee ~]$ rucio upload --rse EULAKE-1 --lifetime 3600 --scope SKA_SKAO_BARNSLEY-testing SKA_SKAO_BARNSLEY-testing:02122021 02122021.1 
2021-12-02 11:49:57,405	INFO	Preparing upload for file 02122021.1
2021-12-02 11:49:57,800	INFO	Successfully added replica in Rucio catalogue at EULAKE-1
2021-12-02 11:49:58,727	INFO	Trying upload with gsiftp to EULAKE-1
2021-12-02 11:50:00,488	INFO	Successful upload of temporary file. gsiftp://eulakeftp.cern.ch:2811/eos/eulake/tests/rucio_test/eulake_1/SKA_SKAO_BARNSLEY-testing/77/32/02122021.1.rucio.upload
2021-12-02 11:50:00,721	INFO	Successfully uploaded file 02122021.1

[user@903de6bd49ee ~]$ rucio list-content SKA_SKAO_BARNSLEY-testing:02122021
+--------------------------------------+--------------+
| SCOPE:NAME                           | [DID TYPE]   |
|--------------------------------------+--------------|
| SKA_SKAO_BARNSLEY-testing:02122021.1 | FILE         |
+--------------------------------------+--------------+

[user@903de6bd49ee ~]$ rucio list-rules SKA_SKAO_BARNSLEY-testing:02122021
ID    ACCOUNT    SCOPE:NAME    STATE[OK/REPL/STUCK]    RSE_EXPRESSION    COPIES    EXPIRES (UTC)    CREATED (UTC)
----  ---------  ------------  ----------------------  ----------------  --------  ---------------  ---------------

[user@903de6bd49ee ~]$ rucio list-rules SKA_SKAO_BARNSLEY-testing:02122021.1
ID    ACCOUNT    SCOPE:NAME    STATE[OK/REPL/STUCK]    RSE_EXPRESSION    COPIES    EXPIRES (UTC)    CREATED (UTC)
----  ---------  ------------  ----------------------  ----------------  --------  ---------------  ---------------

[user@903de6bd49ee ~]$ rucio list-file-replicas SKA_SKAO_BARNSLEY-testing:02122021.1
+---------------------------+------------+------------+-----------+--------------------------------------------------------------------------------------------------------------------------+
| SCOPE                     | NAME       | FILESIZE   | ADLER32   | RSE: REPLICA                                                                                                             |
|---------------------------+------------+------------+-----------+--------------------------------------------------------------------------------------------------------------------------|
| SKA_SKAO_BARNSLEY-testing | 02122021.1 | 2.000 B    | 006e003c  | EULAKE-1: root://eoseulake.cern.ch:1094//eos/eulake/tests/rucio_test/eulake_1/SKA_SKAO_BARNSLEY-testing/77/32/02122021.1 |
+---------------------------+------------+------------+-----------+--------------------------------------------------------------------------------------------------------------------------+
```

compared with the usual behaviour when not specifying this parent dataset:

```
[user@903de6bd49ee ~]$ rucio upload --rse EULAKE-1 --lifetime 3600 --scope SKA_SKAO_BARNSLEY-testing 02122021.2 
2021-12-02 12:05:23,927	INFO	Preparing upload for file 02122021.2
2021-12-02 12:05:24,197	INFO	Successfully added replica in Rucio catalogue at EULAKE-1
2021-12-02 12:05:24,320	INFO	Successfully added replication rule at EULAKE-1
2021-12-02 12:05:24,816	INFO	Trying upload with gsiftp to EULAKE-1
2021-12-02 12:05:26,364	INFO	Successful upload of temporary file. gsiftp://eulakeftp.cern.ch:2811/eos/eulake/tests/rucio_test/eulake_1/SKA_SKAO_BARNSLEY-testing/43/0f/02122021.2.rucio.upload
2021-12-02 12:05:26,611	INFO	Successfully uploaded file 02122021.2

[user@903de6bd49ee ~]$ rucio list-rules SKA_SKAO_BARNSLEY-testing:02122021.2
ID                                ACCOUNT      SCOPE:NAME                            STATE[OK/REPL/STUCK]    RSE_EXPRESSION    COPIES    EXPIRES (UTC)        CREATED (UTC)
--------------------------------  -----------  ------------------------------------  ----------------------  ----------------  --------  -------------------  -------------------
6f6b6dfd8434450ba976498612471b94  robbarnsley  SKA_SKAO_BARNSLEY-testing:02122021.2  OK[1/0/0]               EULAKE-1          1         2021-12-02 13:05:24  2021-12-02 12:05:24
```

which verbosely states that a replication rule has been created.

I'd previously (erroneously) attributed this behaviour to uploading directories, which I think I mentioned during the dev meeting.



Discussion:
I think this is correct. If you upload with a dataset, the rule is created for the dataset, not for individual files.

What Rucio does in the case of upload with a dataset it:
1. Create the dataset
2. Create a rule
3. Uploads the files
4. Attaches the files to the dataset (thus they are covered by the rule)

Could it be that in your workflow, you create the dataset beforehand yourself? In that case rucio will not do step 1 and 2, since the dataset is already there. If it does not have a rule associated, just nothing happens.
Thanks Martin. You are correct, these are existing datasets that these files are being attached to.

Would it not be better to either a) return an error if `--lifetime` is set and the parent dataset already exists, or b) be verbose about the fact it has not created a rule?
There is already a message being displayed
https://github.com/rucio/rucio/blob/226a9f8efb0dfe09b52607cd2333e184ab88d105/lib/rucio/client/uploadclient.py#L403
Although this is only debug, should probably be INFO at least and also give reference that no rule will be created.

I am a bit conflicted, because you could argue that giving `--lifetime` should really enforce a lifetime. But this is quite dangerous on already existing datasets.
imho enforcing a lifetime seems like a side effect. I wouldn't expect an upload function to modify rules on existing data.

Elevating that log level and being more descript about no rule being created may be sufficient, and would've certainly helped me, but as it stands I still think it's ambiguous. If I set the `lifetime` parameter (or any for that matter) I would expect it to a) be used as expected, b) verbosely told that it's being disregarded or c) the program terminated without altering state because the command i've given it doesn't make sense.
I agree.
I think if the dataset already exists, and `--lifetime` is given either b) or c) should happen, with a tendency to c)
@mlassnig @rcarpa opinions? 
I would prefer C) . If the option doesn't make sense, there must be an error and no work done.
I can see where @robbarnsley is coming from. It's certainly very convenient to expect this behaviour.

However, I do have to err on the side of caution here.. it can really cause some unintended effects, so I'd also rather argue for C here.
The only complication I see with this is that people like to repeat commands.
eg they use the upload command once, the upload goes and fails at some file. Often they will just repeat the command since Rucio will resume with the files which failed. In this workflow, that wouldn't work anymore since the dataset exists now and `--lifetime` would block it.
But I think that's ok.

Resolving patch:
diff --git a/lib/rucio/client/uploadclient.py b/lib/rucio/client/uploadclient.py
--- a/lib/rucio/client/uploadclient.py
+++ b/lib/rucio/client/uploadclient.py
@@ -376,7 +376,11 @@ def _register_file(self, file, registered_dataset_dids, ignore_availability=Fals
                                                 'lifetime': file.get('lifetime')}])
                 logger(logging.INFO, 'Successfully created dataset %s' % dataset_did_str)
             except DataIdentifierAlreadyExists:
-                logger(logging.DEBUG, 'Dataset %s already exists' % dataset_did_str)
+                logger(logging.INFO, 'Dataset %s already exists - no rule will be created' % dataset_did_str)
+
+                if file.get('lifetime') is not None:
+                    raise InputValidationError('Dataset %s exists and lifetime %s given. Prohibited to modify parent dataset lifetime.' % (dataset_did_str,
+                                                                                                                                           file.get('lifetime')))
         else:
             logger(logging.DEBUG, 'Skipping dataset registration')
 


Test patch: