Repository: getredash/redash

Issue:
[Feature Request] Update the Redash webhook payload to be a Snowplow-compatible JSON
This is a proposal to take the existing Redash webhook and make some minor changes to make it compatible with [Snowplow](https://github.com/snowplow/snowplow).

As well as allowing a Redash+Snowplow user to warehouse all of their Redash activity in Snowplow for audit and compliance, this has the additional benefit that it ties the Redash webhook to an agreed [JSON Schema format](http://json-schema.org/).

### JSON Schema

We have published a JSON Schema for the Redash webhook as part of Iglu Central, here:

https://github.com/snowplow/iglu-central/blob/master/schemas/io.redash.webhooks/event/jsonschema/1-0-0

The exact naming of the webhook schema (`io.redash.webhooks/event`) was checked with @arikfr and should allow Redash to add further more-specific webhooks in the future as needed.

The schema has been carefully specified so that the generated Redshift table maps very closely onto Redash's underlying event table:

https://github.com/snowplow/iglu-central/blob/master/sql/io.redash.webhooks/event_1.sql

### Steps to make the Redash webhook compatible

Currently the Redash webhook is a POST request with the event data encoded in the body as form data.

We want to change this to JSON, but specifically Iglu-compatible self-describing JSON (Snowplow's schema system is called [Iglu](https://github.com/snowplow/iglu)).

In place of the current POST payload, the Redash webhook should look like this:

```json
{
  "schema": "iglu:io.redash.webhooks/event/jsonschema/1-0-0",
  "data": {
    "id": 23,
    "user_id": null,
    "action": "DELETE",
    ...
  }
}
```

Notes:

* The event's integer types should be kept as numerics in JSON
* The `created_at` field should be a valid ISO8601 date-time

We can easily check any test webhook JSON against the JSON Schema to verify that they match (if there is an error in the JSON Schema, we can patch it in a new Iglu Central release).

### Outcome

When this has been implemented, it will be possible to warehouse all Redash webhooks in Snowplow **with zero coding**, using Snowplow's built-in [Iglu webhook adapter](https://github.com/snowplow/snowplow/wiki/Iglu-webhook-adapter).

There are some really interesting use cases here:

* Full audit log for Redash in Snowplow (allowing the events table in Redash to be regularly truncated?)
* Real-time detection (ML-driven?) of a bad actor based on their SQL queries (e.g. PII theft, corporate espionage)



Discussion:

Resolving patch:
diff --git a/redash/models.py b/redash/models.py
--- a/redash/models.py
+++ b/redash/models.py
@@ -1316,7 +1316,7 @@ class Event(db.Model):
     action = Column(db.String(255))
     object_type = Column(db.String(255))
     object_id = Column(db.String(255), nullable=True)
-    additional_properties = Column(db.Text, nullable=True)
+    additional_properties = Column(MutableDict.as_mutable(PseudoJSON), nullable=True, default={})
     created_at = Column(db.DateTime(True), default=db.func.now())
 
     __tablename__ = 'events'
@@ -1324,6 +1324,17 @@ class Event(db.Model):
     def __unicode__(self):
         return u"%s,%s,%s,%s" % (self.user_id, self.action, self.object_type, self.object_id)
 
+    def to_dict(self):
+        return {
+            'org_id': self.org_id,
+            'user_id': self.user_id,
+            'action': self.action,
+            'object_type': self.object_type,
+            'object_id': self.object_id,
+            'additional_properties': self.additional_properties,
+            'created_at': self.created_at.isoformat()
+        }
+
     @classmethod
     def record(cls, event):
         org_id = event.pop('org_id')
@@ -1333,11 +1344,10 @@ def record(cls, event):
         object_id = event.pop('object_id', None)
 
         created_at = datetime.datetime.utcfromtimestamp(event.pop('timestamp'))
-        additional_properties = json.dumps(event)
 
         event = cls(org_id=org_id, user_id=user_id, action=action,
                     object_type=object_type, object_id=object_id,
-                    additional_properties=additional_properties,
+                    additional_properties=event,
                     created_at=created_at)
         db.session.add(event)
         return event
diff --git a/redash/tasks/general.py b/redash/tasks/general.py
--- a/redash/tasks/general.py
+++ b/redash/tasks/general.py
@@ -10,14 +10,18 @@
 
 
 @celery.task(name="redash.tasks.record_event", base=BaseTask)
-def record_event(event):
-    original_event = event.copy()
-    models.Event.record(event)
+def record_event(raw_event):
+    event = models.Event.record(raw_event)
     models.db.session.commit()
+
     for hook in settings.EVENT_REPORTING_WEBHOOKS:
         logger.debug("Forwarding event to: %s", hook)
         try:
-            response = requests.post(hook, original_event)
+            data = {
+              "schema": "iglu:io.redash.webhooks/event/jsonschema/1-0-0",
+              "data": event.to_dict()
+            }
+            response = requests.post(hook, json=data)
             if response.status_code != 200:
                 logger.error("Failed posting to %s: %s", hook, response.content)
         except Exception:


Test patch:
diff --git a/tests/test_models.py b/tests/test_models.py
--- a/tests/test_models.py
+++ b/tests/test_models.py
@@ -409,7 +409,7 @@ def test_records_additional_properties(self):
 
         event = models.Event.record(raw_event)
 
-        self.assertDictEqual(json.loads(event.additional_properties), additional_properties)
+        self.assertDictEqual(event.additional_properties, additional_properties)
 
 
 class TestWidgetDeleteInstance(BaseTestCase):