Repository: hydroshare/hydroshare

Issue:
Geoserver update_web_services async task is failing
**Description of the bug**
`update_web_services` Celery task is failing (and seems to have been for a while).
This function sends a resource id to the HydroShare web services manager
    application. Seems there is a key error and the Geoserver is not returning a json response

Steps to reproduce the bug:
Navigate to the celery worker task UI on production and search for the update_web_services task.


**Expected behavior**
Celery task should finish correctly. Geoserver should return a json serializable response.

**Additional information**
Add any other context about the problem here such as screenshots, software versions, and browser information.
https://github.com/hydroshare/hydroshare/blob/develop/hs_core/tasks.py#L751

Example log snippet:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/json.py", line 70, in dumps
    **dict(default_kwargs, **kwargs))
  File "/usr/local/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/usr/local/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'KeyError' is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 449, in trace_task
    uuid, retval, task_request, publish_result,
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 149, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/amqp.py", line 140, in store_result
    delivery_mode=self.delivery_mode,
  File "/usr/local/lib/python3.6/site-packages/kombu/messaging.py", line 169, in publish
    compression, headers)
  File "/usr/local/lib/python3.6/site-packages/kombu/messaging.py", line 252, in _prepare
    body) = dumps(body, serializer=serializer)
  File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py", line 54, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/vine/five.py", line 194, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/usr/local/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/json.py", line 70, in dumps
    **dict(default_kwargs, **kwargs))
  File "/usr/local/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/usr/local/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
kombu.exceptions.EncodeError: Object of type 'KeyError' is not JSON serializable
```



Discussion:
@martinseul and I took a look at this,
confirmed that the [update_web_services](https://github.com/hydroshare/hydroshare/blob/develop/hs_core/tasks.py#L634) function is indeed failing...
![image](https://user-images.githubusercontent.com/17934193/202021031-173a2282-1857-41c5-9599-e6cb8097eb4a.png)
was curious so checked again just now. Yes, still failing.
@martinseul making sure you're aware that this is still failing.
@sblack-usu and I took a look
This issue arises because the hs_data_services app doesn't return any content:
https://github.com/hydroshare/hs_data_services/blob/main/hs_data_services/hs_data_services_sync/views.py#L29

So HS doesnt have any content to update with
https://github.com/hydroshare/hydroshare/blob/master/hs_core/tasks.py#L681
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/usr/local/lib/python3.9/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.9/site-packages/kombu/utils/json.py", line 69, in dumps
    return _dumps(s, cls=cls or _default_encoder,
  File "/usr/local/lib/python3.9/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/local/lib/python3.9/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.9/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.9/site-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/usr/local/lib/python3.9/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Response is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 479, in trace_task
    mark_as_done(
  File "/usr/local/lib/python3.9/site-packages/celery/backends/base.py", line 158, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File "/usr/local/lib/python3.9/site-packages/celery/backends/amqp.py", line 132, in store_result
    producer.publish(
  File "/usr/local/lib/python3.9/site-packages/kombu/messaging.py", line 167, in publish
    body, content_type, content_encoding = self._prepare(
  File "/usr/local/lib/python3.9/site-packages/kombu/messaging.py", line 252, in _prepare
    body) = dumps(body, serializer=serializer)
  File "/usr/local/lib/python3.9/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/kombu/serialization.py", line 54, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/usr/local/lib/python3.9/site-packages/vine/five.py", line 194, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.9/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/usr/local/lib/python3.9/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.9/site-packages/kombu/utils/json.py", line 69, in dumps
    return _dumps(s, cls=cls or _default_encoder,
  File "/usr/local/lib/python3.9/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/local/lib/python3.9/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.9/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.9/site-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/usr/local/lib/python3.9/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
kombu.exceptions.EncodeError: Object of type Response is not JSON serializable
```

Resolving patch:
diff --git a/hs_core/tasks.py b/hs_core/tasks.py
--- a/hs_core/tasks.py
+++ b/hs_core/tasks.py
@@ -779,35 +779,26 @@ def update_web_services(services_url, api_token, timeout, publish_urls, res_id):
         response = session.post(rest_url, timeout=timeout)
 
         if publish_urls and response.status_code == status.HTTP_201_CREATED:
-            try:
-
-                resource = utils.get_resource_by_shortkey(res_id)
-                response_content = json.loads(response.content.decode())
-
-                if "resource" in response_content:
-                    for key, value in response_content["resource"].items():
-                        resource.extra_metadata[key] = value
-                        resource.save()
-
-                if "content" in response_content:
-                    for url in response_content["content"]:
-                        logical_files = list(resource.logical_files)
-                        lf = logical_files[[i.aggregation_name for i in
-                                            logical_files].index(
-                            url["layer_name"].encode()
-                        )]
-                        lf.metadata.extra_metadata["Web Services URL"] = url["message"]
-                        lf.metadata.save()
-
-            except Exception as e:
-                logger.error(e)
-                return e
-
-        return response
-
-    except (requests.exceptions.RequestException, ValueError) as e:
-        logger.error(e)
-        return e
+            resource = utils.get_resource_by_shortkey(res_id)
+            response_content = json.loads(response.content.decode())
+            if "resource" in response_content:
+                for key, value in response_content["resource"].items():
+                    resource.extra_metadata[key] = value
+                    resource.save()
+
+            if "content" in response_content:
+                for url in response_content["content"]:
+                    logical_files = list(resource.logical_files)
+                    lf = logical_files[[i.aggregation_name for i in
+                                        logical_files].index(
+                        url["layer_name"].encode()
+                    )]
+                    lf.metadata.extra_metadata["Web Services URL"] = url["message"]
+                    lf.metadata.save()
+        return response.json()
+    except Exception as e:
+        logger.error(f"Error updating web services: {str(e)}")
+        raise
 
 
 @shared_task


Test patch: