Repository: rucio/rucio

Issue:
Expose request details (e.g. FTS job ID) to the client
Motivation
----------
Clients should be able to retrieve request details (FTS jobID etc) for transfer monitoring purpose.

Modification
------------
Make results available in query_request (@api/request.py) into web/rest/request.py. 
If I have understood well the suggestion by @mlassnig .


Discussion:
@dciangot 
Which access pattern would you see to retrieve this information ?  `requestId` is very internal to Rucio and exposed at few places. Drafting the corresponding REST APIs and workflow would help.


@vingar  In fact the idea is that a system using Rucio clients (e.g. CRAB for CMS user data) wants to keep track of the FTS job ID/IDs for a certain DID against a user defined RSE, only for debugging purpose at the moment.

In this sense I think the access pattern could be as complicated as it needs to be, e.g. first retrieve request ID for a DID and then to a second call to retrieve the request details.
An optimal solution instead, but probably too specific for the use case, would be that the details of the last request will be shown providing a set of (did, source_rse, destination_rse).

Does it make any sense for you?

P.S. using the MQ would be a possibilty too for this problem, but I fill like it is a general-enough problem that one may want to pass it directly through the client.
yes, it makes perfect sense.  I think the workflow will be from DID, RSE -> Rule -> RequestIds -> FTS jobs Ids.

Few things are already there at least in the rucio UI + rules CLIs for this purpose. It should be checked. 
For what I have understood the only missing part there is RequestIds -> FTS jobs, at list from CLI point of view.

So a minimal solution here is essentially the porting of  RequestIds -> FTS jobs step into CLI/python bindings from where it is already available in Rucio core. At least this is my understanding/opinion at the moment.
I had a look at the UI code showing the status of a rule (https://github.com/rucio/rucio/blob/master/lib/rucio/web/ui/static/rule.js) and there it calls `get_request_by_did` which corresponds to the REST call:

     https://github.com/rucio/rucio/blob/master/lib/rucio/web/rest/webpy/v1/request.py

I have the feeling that's what you need. Maybe some convenient clients and wrappers missing.

Yes, I think that what we are looking for is to integrate the info of `get_request_by_did` with the info coming from `query_request` (*) in some form.

(*) https://github.com/rucio/rucio/blob/next/lib/rucio/core/request.py#L333
mmm... btw, probably `get_request_by_did` it is enough alone. If I understand well the query made there. So probably it is really something minor that is missing. 
But I think that it is now more or less clear what I have to check.
So, summarizing with a working plan: 
1. add a call here (*), or in another client if you prefer, like:
 ``` def get_request_by_did(self, name, rse, scope=None)```
this will use the rest call to: ```https://github.com/rucio/rucio/blob/master/lib/rucio/web/rest/webpy/v1/request.py```

2. Problem is that I have just checked that due to this (**), the request rest call is broken for CMS DIDs (containing `/`). So this is to be fixed before`1.` (that is pretty straight forward) can work.

How does it sound? Did I miss anything?

(*) https://github.com/rucio/rucio/blob/next/lib/rucio/client/didclient.py
(**) https://github.com/rucio/rucio/blob/master/lib/rucio/web/rest/webpy/v1/request.py#L38
About 1. 
With the current per resource logic, it should go in a new rucio/client/requestclient.py file.

2. looks fixable ;)
> About 1.
> With the current per resource logic, it should go in a new rucio/client/requestclient.py file.
> 

ok then, I'll put in `rucio/client/requestclient.py`

> 2. looks fixable ;)

yeah, indeed :) do you prefer a separate issue for that?


Hi, yes, agree on a new `requestclient`. For 2 a new ticket would be helpful, thanks.
I think a new requestclient would be good as well. The concern I see a bit, also what Vincent said, that previously the requestId was a rather internal concept, but since we are exposing it already via the REST (For the WebUI fts-id linking), I think it is good to expose it properly via a client. Also it's a proper usecase to be able to access fts-ids for external monitoring.
from the described workflow, it looks not needed to interact with `requestId`
Yes you are right, it's not interacting, but exposing (primarily for the fts-id).

Resolving patch:
diff --git a/lib/rucio/client/client.py b/lib/rucio/client/client.py
--- a/lib/rucio/client/client.py
+++ b/lib/rucio/client/client.py
@@ -33,6 +33,7 @@
 from rucio.client.metaclient import MetaClient
 from rucio.client.pingclient import PingClient
 from rucio.client.replicaclient import ReplicaClient
+from rucio.client.requestclient import RequestClient
 from rucio.client.rseclient import RSEClient
 from rucio.client.ruleclient import RuleClient
 from rucio.client.scopeclient import ScopeClient
@@ -41,7 +42,7 @@
 from rucio.client.touchclient import TouchClient
 
 
-class Client(AccountClient, AccountLimitClient, MetaClient, PingClient, ReplicaClient, RSEClient, ScopeClient, DIDClient, RuleClient, SubscriptionClient, LockClient, ConfigClient, TouchClient):
+class Client(AccountClient, AccountLimitClient, MetaClient, PingClient, ReplicaClient, RequestClient, RSEClient, ScopeClient, DIDClient, RuleClient, SubscriptionClient, LockClient, ConfigClient, TouchClient):
 
     """Main client class for accessing Rucio resources. Handles the authentication."""
 
diff --git a/lib/rucio/client/requestclient.py b/lib/rucio/client/requestclient.py
new file mode 100644
--- /dev/null
+++ b/lib/rucio/client/requestclient.py
@@ -0,0 +1,61 @@
+# Copyright 2013-2018 CERN for the benefit of the ATLAS collaboration.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Authors:
+# - Diego Ciangottini <diego.ciangottini@gmail.com>, 2018
+
+try:
+    from urllib import quote_plus
+except ImportError:
+    from urllib.parse import quote_plus
+
+from requests.status_codes import codes
+
+from rucio.client.baseclient import BaseClient
+from rucio.client.baseclient import choice
+from rucio.common.utils import build_url
+
+
+class RequestClient(BaseClient):
+
+    REQUEST_BASEURL = 'requests'
+
+    def __init__(self, rucio_host=None, auth_host=None, account=None, ca_cert=None,
+                 auth_type=None, creds=None, timeout=600, user_agent='rucio-clients'):
+        super(RequestClient, self).__init__(rucio_host, auth_host, account, ca_cert,
+                                            auth_type, creds, timeout, user_agent)
+
+    def list_request_by_did(self, name, rse, scope=None):
+        """Return latest request details for a DID
+
+        :param name: DID
+        :type name: str
+        :param rse: Destination RSE name
+        :type rse: str
+        :param scope: rucio scope, defaults to None
+        :param scope: str, optional
+        :raises exc_cls: from BaseClient._get_exception
+        :return: request information
+        :rtype: dict
+        """
+
+        path = '/'.join(['requests', quote_plus(scope), quote_plus(name), rse])
+        url = build_url(choice(self.list_hosts), path=path)
+        r = self._send_request(url, type='GET')
+
+        if r.status_code == codes.ok:
+            return next(self._load_json_data(r))
+        else:
+            exc_cls, exc_msg = self._get_exception(headers=r.headers, status_code=r.status_code, data=r.content)
+            raise exc_cls(exc_msg)


Test patch: