Repository: hydroshare/hydroshare

Issue:
Geoserver update_web_services async task is failing
**Description of the bug**
`update_web_services` Celery task is failing (and seems to have been for a while).
This function sends a resource id to the HydroShare web services manager
    application. Seems there is a key error and the Geoserver is not returning a json response

Steps to reproduce the bug:
Navigate to the celery worker task UI on production and search for the update_web_services task.


**Expected behavior**
Celery task should finish correctly. Geoserver should return a json serializable response.

**Additional information**
Add any other context about the problem here such as screenshots, software versions, and browser information.
https://github.com/hydroshare/hydroshare/blob/develop/hs_core/tasks.py#L751

Example log snippet:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/json.py", line 70, in dumps
    **dict(default_kwargs, **kwargs))
  File "/usr/local/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/usr/local/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'KeyError' is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 449, in trace_task
    uuid, retval, task_request, publish_result,
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 149, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/amqp.py", line 140, in store_result
    delivery_mode=self.delivery_mode,
  File "/usr/local/lib/python3.6/site-packages/kombu/messaging.py", line 169, in publish
    compression, headers)
  File "/usr/local/lib/python3.6/site-packages/kombu/messaging.py", line 252, in _prepare
    body) = dumps(body, serializer=serializer)
  File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py", line 54, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/vine/five.py", line 194, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/json.py", line 70, in dumps
    **dict(default_kwargs, **kwargs))
  File "/usr/local/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/usr/local/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
kombu.exceptions.EncodeError: Object of type 'KeyError' is not JSON serializable
```



Discussion:
@martinseul and I took a look at this,
confirmed that the [update_web_services](https://github.com/hydroshare/hydroshare/blob/develop/hs_core/tasks.py#L634) function is indeed failing...
![image](https://user-images.githubusercontent.com/17934193/202021031-173a2282-1857-41c5-9599-e6cb8097eb4a.png)
was curious so checked again just now. Yes, still failing.
@martinseul making sure you're aware that this is still failing.
@sblack-usu and I took a look
This issue arises because the hs_data_services app doesn't return any content:
https://github.com/hydroshare/hs_data_services/blob/main/hs_data_services/hs_data_services_sync/views.py#L29

So HS doesnt have any content to update with
https://github.com/hydroshare/hydroshare/blob/master/hs_core/tasks.py#L681

Resolving patch:
diff --git a/hs_core/tasks.py b/hs_core/tasks.py
--- a/hs_core/tasks.py
+++ b/hs_core/tasks.py
@@ -750,18 +750,20 @@ def update_web_services(services_url, api_token, timeout, publish_urls, res_id):
                 resource = utils.get_resource_by_shortkey(res_id)
                 response_content = json.loads(response.content.decode())
 
-                for key, value in response_content["resource"].items():
-                    resource.extra_metadata[key] = value
-                    resource.save()
-
-                for url in response_content["content"]:
-                    logical_files = list(resource.logical_files)
-                    lf = logical_files[[i.aggregation_name for i in
-                                        logical_files].index(
-                        url["layer_name"].encode()
-                    )]
-                    lf.metadata.extra_metadata["Web Services URL"] = url["message"]
-                    lf.metadata.save()
+                if "resource" in response_content:
+                    for key, value in response_content["resource"].items():
+                        resource.extra_metadata[key] = value
+                        resource.save()
+
+                if "content" in response_content:
+                    for url in response_content["content"]:
+                        logical_files = list(resource.logical_files)
+                        lf = logical_files[[i.aggregation_name for i in
+                                            logical_files].index(
+                            url["layer_name"].encode()
+                        )]
+                        lf.metadata.extra_metadata["Web Services URL"] = url["message"]
+                        lf.metadata.save()
 
             except Exception as e:
                 logger.error(e)


Test patch: