Repository: Flexget/Flexget

Issue:
API "PUT"  method for variables endpoint fails to update variables.yml
### Expected behaviour: The `variables.yml` file to be updated to reflect the json payload sent via curl.

### Actual behaviour: No updates are made.

### Steps to reproduce:
- Step 1: Create `variables.yml` file with only the following single line:

`a:b`

- Step 2: From terminal, execute: `curl -X PUT --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'Authorization: Token TOKEN_REDACTED' -d '{"a":"c"}' 'http://localhost:5050/api/variables/'`

- Step 3: Observe output as:
`a:b`

#### Config:

```
variables: /path/to/flexget/variables.yml
web_server: yes
templates:
  dl:
    deluge: yes
....
...
...
```

#### Variables:
```
a: b
```
#### Log:
<details>
No information was added to the log after the API call.
</details>

### Additional information:

- FlexGet version: 2.21.25  
- Python version: 3.7.3
- Installation method: pip3
- Using daemon (yes/no): yes
- OS and version: Kubuntu 19.04
- Link to crash log: null
API "PUT"  method for variables endpoint fails to update variables.yml
### Expected behaviour: The `variables.yml` file to be updated to reflect the json payload sent via curl.

### Actual behaviour: No updates are made.

### Steps to reproduce:
- Step 1: Create `variables.yml` file with only the following single line:

`a:b`

- Step 2: From terminal, execute: `curl -X PUT --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'Authorization: Token TOKEN_REDACTED' -d '{"a":"c"}' 'http://localhost:5050/api/variables/'`

- Step 3: Observe output as:
`a:b`

#### Config:

```
variables: /path/to/flexget/variables.yml
web_server: yes
templates:
  dl:
    deluge: yes
....
...
...
```

#### Variables:
```
a: b
```
#### Log:
<details>
No information was added to the log after the API call.
</details>

### Additional information:

- FlexGet version: 2.21.25  
- Python version: 3.7.3
- Installation method: pip3
- Using daemon (yes/no): yes
- OS and version: Kubuntu 19.04
- Link to crash log: null


Discussion:
@gazpachoking was able to shed some light on this issue via the slack channel. 

It seems that the `variables.yml` acts as a 'seed' for the variables section in the flexget database, when the line `variables: /path/to/variables.yml` is included in `config.yml`.  The `PUT` method for the flexget API with the `variables` endpoint only works when the that line in `config.yml` is amended to `variables: yes` instead of the location of the `variables.yml` file.

However, using the command:

```curl -X PUT --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'Authorization: Token TOKEN_REDACTED' -d @variables.json 'http://localhost:5050/api/variables/'```

results in the the variables being wiped out and replaced by the JSON payload (the `@variables.json` in the above command).

I have a very long list of variables (hundreds). So I would need to use the `GET` method on the variables first, save the output as a `variables.json` file, make the change(s), save it, then use the `PUT` method to reinstall the new variables in the database via the updated `variables.json` file.

Is it possible to use the API to simply change a single value within the variables? Or is that function not available?
The API only operates on the DB and not on the file. This was a design decision as to eliminate having two sources of truth.
@liiight The new issue he was talking about is that sending a PUT replaces the entire variable store. So to update a single variable you have to do a get, modify the returned data structure, then submit the whole thing back. Maybe we should add a PATCH endpoint or something where you can send in just the variables you want updated.
Ah ok. I'm not sure how to do that since variables is just a json blob basically. We have no base structure for it so I'm not sure how we could add single "entity", since that entity's definition can change for each user.
I mean, if we wanted a PATCH, it could just load the current variables, call an update on it and save it again.
I'm not sure how that would work. How would you join the existing data to the new payload?
Oh, you something like do a dict `.update()` using the new payload? Guess that could work 
Yep, a dict.update is exactly what I was thinking.
Hmm, not sure the optimal way PATCH should work for heavily nested variables files. Would it be a recursive dict.update, or just on the first level?
I think first level is good enough.
Although, I maybe something like [glom](https://github.com/mahmoud/glom) would be nice here. I use it heavily at work and it's amazing for operating on nested data (among other things).
If we want the update to work with nested data that's easy enough to do. We could also change the api interface to allow using json pointers to get/change specific nested data.

e.g. PUT /variables/some/nested/path/0
Maybe only doing a first level update with PATCH, and adding the nested access via path is the nicest way.
dunno, I don't really like the path approach. I see variables as a single object, I don't really like to start manipulating it so dynamically. Also, json pointer like that via path is kinda weird to me, never saw something like out in the wild
@gazpachoking was able to shed some light on this issue via the slack channel. 

It seems that the `variables.yml` acts as a 'seed' for the variables section in the flexget database, when the line `variables: /path/to/variables.yml` is included in `config.yml`.  The `PUT` method for the flexget API with the `variables` endpoint only works when the that line in `config.yml` is amended to `variables: yes` instead of the location of the `variables.yml` file.

However, using the command:

```curl -X PUT --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'Authorization: Token TOKEN_REDACTED' -d @variables.json 'http://localhost:5050/api/variables/'```

results in the the variables being wiped out and replaced by the JSON payload (the `@variables.json` in the above command).

I have a very long list of variables (hundreds). So I would need to use the `GET` method on the variables first, save the output as a `variables.json` file, make the change(s), save it, then use the `PUT` method to reinstall the new variables in the database via the updated `variables.json` file.

Is it possible to use the API to simply change a single value within the variables? Or is that function not available?
The API only operates on the DB and not on the file. This was a design decision as to eliminate having two sources of truth.
@liiight The new issue he was talking about is that sending a PUT replaces the entire variable store. So to update a single variable you have to do a get, modify the returned data structure, then submit the whole thing back. Maybe we should add a PATCH endpoint or something where you can send in just the variables you want updated.
Ah ok. I'm not sure how to do that since variables is just a json blob basically. We have no base structure for it so I'm not sure how we could add single "entity", since that entity's definition can change for each user.
I mean, if we wanted a PATCH, it could just load the current variables, call an update on it and save it again.
I'm not sure how that would work. How would you join the existing data to the new payload?
Oh, you something like do a dict `.update()` using the new payload? Guess that could work 
Yep, a dict.update is exactly what I was thinking.
Hmm, not sure the optimal way PATCH should work for heavily nested variables files. Would it be a recursive dict.update, or just on the first level?
I think first level is good enough.
Although, I maybe something like [glom](https://github.com/mahmoud/glom) would be nice here. I use it heavily at work and it's amazing for operating on nested data (among other things).
If we want the update to work with nested data that's easy enough to do. We could also change the api interface to allow using json pointers to get/change specific nested data.

e.g. PUT /variables/some/nested/path/0
Maybe only doing a first level update with PATCH, and adding the nested access via path is the nicest way.
dunno, I don't really like the path approach. I see variables as a single object, I don't really like to start manipulating it so dynamically. Also, json pointer like that via path is kinda weird to me, never saw something like out in the wild

Resolving patch:
diff --git a/flexget/api/app.py b/flexget/api/app.py
--- a/flexget/api/app.py
+++ b/flexget/api/app.py
@@ -21,7 +21,7 @@
 from flexget.webserver import User
 from . import __path__
 
-__version__ = '1.4.3'
+__version__ = '1.5.0'
 
 log = logging.getLogger('api')
 
diff --git a/flexget/components/variables/api.py b/flexget/components/variables/api.py
--- a/flexget/components/variables/api.py
+++ b/flexget/components/variables/api.py
@@ -35,3 +35,18 @@ def put(self, session=None):
         rsp = jsonify(variables_from_db())
         rsp.status_code = 201
         return rsp
+
+    @api.response(200, 'Successfully updated variables file')
+    @api.validate(empty_response)
+    @api.doc(
+        description='Note that editing variables may not be persistent, depending on user config'
+    )
+    def patch(self, session=None):
+        """ Update variables data to DB """
+        data = request.json
+        existing_variables = variables_from_db()
+        existing_variables.update(data)
+        variables_to_db(existing_variables)
+        # This will trigger reloading the variables file
+        self.manager.validate_config()
+        return jsonify(variables_from_db())


Test patch:
diff --git a/flexget/tests/api_tests/test_variables_api.py b/flexget/tests/api_tests/test_variables_api.py
--- a/flexget/tests/api_tests/test_variables_api.py
+++ b/flexget/tests/api_tests/test_variables_api.py
@@ -32,3 +32,12 @@ def test_variables_put(self, api_client):
         rsp = api_client.get('/variables/')
         assert rsp.status_code == 200, 'Response code is %s' % rsp.status_code
         assert json.loads(rsp.get_data(as_text=True)) == self.variables_dict
+
+    def test_variables_patch(self, api_client):
+        data = {'a': 'b', 'c': 'd'}
+        api_client.json_put('/variables/', data=json.dumps(data))
+        new_data = {'a': [1, 2, 3], 'foo': 'bar'}
+
+        rsp = api_client.json_patch('/variables/', data=json.dumps(new_data))
+        assert rsp.status_code == 200, 'Response code is %s' % rsp.status_code
+        assert json.loads(rsp.get_data(as_text=True)) == {'a': [1, 2, 3], 'foo': 'bar', 'c': 'd'}
diff --git a/flexget/tests/conftest.py b/flexget/tests/conftest.py
--- a/flexget/tests/conftest.py
+++ b/flexget/tests/conftest.py
@@ -375,6 +375,12 @@ def json_put(self, *args, **kwargs):
             self._append_header('Authorization', 'Token %s' % self.api_key, kwargs)
         return self.client.put(*args, **kwargs)
 
+    def json_patch(self, *args, **kwargs):
+        self._append_header('Content-Type', 'application/json', kwargs)
+        if kwargs.get('auth', True):
+            self._append_header('Authorization', 'Token %s' % self.api_key, kwargs)
+        return self.client.patch(*args, **kwargs)
+
     def get(self, *args, **kwargs):
         if kwargs.get('auth', True):
             self._append_header('Authorization', 'Token %s' % self.api_key, kwargs)