giseldo/neodataset huggingface dataset:
Expose Engine Endpoint for Querying Nostr Events

<!---
Please read this!

The Definition of Ready (DoR) can be found at https://developers.minds.com/docs/handbook/how-we-work/#definition-of-ready-dor

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "Type::Feature" label and verify the feature you're about to submit isn't a duplicate.
--->

## Goal

<!---
Clearly outline the goal of the issue and why we are doing it
--->

Currently we expose the `fetch-entity` endpoint that allows for retrieving Nostr events based on the event id, although we should extend this to allow for more robust querying based on the following fields:

* id
* author
* `e` and `p` tags
* date range

**Note**: This index exists in Sandbox with the following mapping:

`minds-nostr-event-v1`

```json
{
  "mappings": {
    "properties": {
      "id": { "type": "keyword" },
      "pubkey": { "type": "keyword" },
      "created_at": { "type": "date" },
      "kind": { "type": "short" },
      "tags": { "type": "keyword" },
      "sig": { "type": "keyword" }
    }
  }
}
```

## What needs to be done

<!---
Clearly detail what needs to be done to achieve the goal set out above
--->

1. Create an Elasticsearch index for containing Nostr events.
  * Do not include the `content` of the events, just the metadata
2. Setup a runner for syncing created entities to the events index.
3. Expose an endpoint for returning the formatted Nostr events that match the above fields 

## QA

<!---
Consideration is given to how the issue will be demoed and tested prior to being deployed
--->

We can test by deploying to Sandbox alongside [our Nostr relay](https://gitlab.com/minds/infrastructure/nostr-relay).

## UX/Design

<!---
Sufficient wireframes have been presented and understood by the development team. Full mockups are not required and re-usable components are encouraged.
--->

N/A

## Personas

<!---
Who does this user impact and why (optional)
--->

N/A

## Experiments

<!---
Experiments are not required, but it should be made clear if one is expected. If experiments are required, the hypothesis should be outlined.
--->

N/A

## Acceptance Criteria

<!--
Everyone should be able to understand what is expected to be delivered
--->

- [ ] Nostr events can be queried using the above endpoint

## Definition of Ready Checklist

- [X] Definition Of Done (DoD)
- [X] Acceptance criteria
- [ ] Weighted
- [X] QA
- [X] UX/Design
- [X] Personas
- [X] Experiments

Story points: 3.0
Sync a users subscriptions to new 'Vitess' database

<!---
Please read this!

The Definition of Ready (DoR) can be found at https://developers.minds.com/docs/handbook/how-we-work/#definition-of-ready-dor

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "Type::Feature" label and verify the feature you're about to submit isn't a duplicate.
--->

## Goal

<!---
Clearly outline the goal of the issue and why we are doing it
--->

Cassandra, ElasticSearch and Redis do not allow us to do simple join queries. This makes 'mutual friends' and other such queries very difficult. [Vitess](https://vitess.io/) is a CloudNative project that brings high availability to MySQL.  

## What needs to be done

<!---
Clearly detail what needs to be done to achieve the goal set out above
--->

We already have an event stream for all subscribe actions 'event-action-subscribe'. We can tab into this and keep in sync with Vitess via a background runner. 

Prior to this, we can import a list of all 'friends' (subscriptions) by dumping a CSV from Cassandra and importing into Vitess. 

## QA

<!---
Consideration is given to how the issue will be demoed and tested prior to being deployed
--->

N/A

## UX/Design

<!---
Sufficient wireframes have been presented and understood by the development team. Full mockups are not required and re-usable components are encouraged.
--->

N/A

## Personas

<!---
Who does this user impact and why (optional)
--->

Devs

## Experiments

<!---
Experiments are not required, but it should be made clear if one is expected. If experiments are required, the hypothesis should be outlined.
--->

N/A

## Acceptance Criteria

<!--
Everyone should be able to understand what is expected to be delivered
--->

- [ ] Update minds/php image to support PDO driver
- [ ] A new `Data\SQL\Client` that will point to our Vitess cluster
- [ ] Schemas for our 'friends' table
- [ ] A background runner that subscribes to 'event-action-subscribe' and sync to our SQL friends table
- [ ] Manual bulk job to copy from cassandra to vitess
- [ ] Documentation

## Definition of Ready Checklist

- [x] Definition Of Done (DoD)
- [x] Acceptance criteria
- [x] Weighted
- [x] QA
- [x] UX/Design
- [x] Personas
- [x] Experiments

Story points: 13.0
Remove legacy helpdesk

<!---
Please read this!

The Definition of Ready (DoR) can be found at https://developers.minds.com/docs/handbook/how-we-work/#definition-of-ready-dor

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "Type::Feature" label and verify the feature you're about to submit isn't a duplicate.
--->

## Goal

<!---
Clearly outline the goal of the issue and why we are doing it
--->
Housekeeping -- remove references to legacy helpdesk.

## What needs to be done

<!---
Clearly detail what needs to be done to achieve the goal set out above
--->
Delete files related to legacy helpdesk.

## QA

<!---
Consideration is given to how the issue will be demoed and tested prior to being deployed
--->
Probably don't need it because the helpdesk isn't being used anymore.

## UX/Design

<!---
Sufficient wireframes have been presented and understood by the development team. Full mockups are not required and re-usable components are encouraged.
--->


## Personas

<!---
Who does this user impact and why (optional)
--->
Developers.

## Experiments

<!---
Experiments are not required, but it should be made clear if one is expected. If experiments are required, the hypothesis should be outlined.
--->
None

## Acceptance Criteria

<!--
Everyone should be able to understand what is expected to be delivered
--->

- [ ] Legacy helpdesk deleted

## Definition of Ready Checklist

- [ ] Definition Of Done (DoD)
- [ ] Acceptance criteria
- [ ] Weighted
- [ ] QA
- [ ] UX/Design
- [ ] Personas
- [ ] Experiments

Story points: 1.0
[Spike] Explore replacing Etherscans use for getting block number by time with TheGraph or BigQuery

<!---
Please read this!

The Definition of Ready (DoR) can be found at https://developers.minds.com/docs/handbook/how-we-work/#definition-of-ready-dor

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "Type::Feature" label and verify the feature you're about to submit isn't a duplicate.
--->

## Goal

Cut down our dependence on external services by seeing if we can migrate our use of Etherscan to get the current block number by timestamp (for example in rewards calculation) to using TheGraph, BigQuery, or possibly Infura. As far as I'm aware to do this without an external dependency, we would need to iterate through all previous blocks in reverse order which is an expensive operation - to do this engine side we'd also be hitting Infura if using the web3 service anyway - unless we set up our own Ethereum node. Ideally, we can do this in one sweep with BigQuery or TheGraph.

Possible resources to explore
- https://thegraph.com/explorer/subgraph?id=3WFXNz46rk4iuVgsBybcGtxMa4cbHkBLfuSjUvvqs2MD&view=Overview
- https://cloud.google.com/blog/products/data-analytics/ethereum-bigquery-public-dataset-smart-contract-analytics

## What needs to be done

Explore whether we can get the closest block number to a timestamp using another service that we already use, cutting down the amount of third parties that we depend on.

## QA

Will need to test block number is calculated correctly for holding rewards

## UX/Design

N/A

## Personas

- Engagement reward enrolled users - by cutting the chances of downtime. 
- Team - make it simpler to use less APIs

## Experiments

N/A

## Acceptance Criteria

- [ ] Get a prototype together of a query/endpoint that can fetch this
- [ ] Possibly build out integration (to discuss)

## Definition of Ready Checklist

- [x] Definition Of Done (DoD)
- [x] Acceptance criteria
- [ ] Weighted
- [x] QA
- [x] UX/Design
- [x] Personas
- [x] Experiments

xplo

Story points: 3.0
Create a read-only nostr relay for Minds posts

<!---
Please read this!

The Definition of Ready (DoR) can be found at https://developers.minds.com/docs/handbook/how-we-work/#definition-of-ready-dor

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "Type::Feature" label and verify the feature you're about to submit isn't a duplicate.
--->

## Goal

<!---
Clearly outline the goal of the issue and why we are doing it
--->

Following on from https://gitlab.com/minds/engine/-/issues/2325, where we **push** out to other Nostr relays, we discovered that the network would become saturated with Minds posts and overwhelm the other Nostr community relay nodes. https://gitlab.com/minds/engine/-/issues/2332 will allow us **pull** Minds posts and channels from Nostr ID (hashes). 

## What needs to be done

<!---
Clearly detail what needs to be done to achieve the goal set out above
--->

PHP and websockets don't play together nicely. Instead, consider a simple NodeJS socket server that proxies the endpoint returned in https://gitlab.com/minds/engine/-/issues/2332.

## QA

<!---
Consideration is given to how the issue will be demoed and tested prior to being deployed
--->

Minds posts can be returned from a Nostr client. 

## UX/Design

<!---
Sufficient wireframes have been presented and understood by the development team. Full mockups are not required and re-usable components are encouraged.
--->

N/A


## Personas

<!---
Who does this user impact and why (optional)
--->

Digital Nomads

## Experiments

<!---
Experiments are not required, but it should be made clear if one is expected. If experiments are required, the hypothesis should be outlined.
--->


## Acceptance Criteria

<!--
Everyone should be able to understand what is expected to be delivered
--->

- [ ] A simple nodejs that proxies the https://gitlab.com/minds/engine/-/issues/2332 endpoint
- [ ] Helm chart
- [ ] Spec tests (bonus)

## Definition of Ready Checklist

- [ ] Definition Of Done (DoD)
- [ ] Acceptance criteria
- [ ] Weighted
- [ ] QA
- [ ] UX/Design
- [ ] Personas
- [ ] Experiments

Story points: 5.0
Implement reverse index for Nostr hashes (event ids) to Minds posts

<!---
Please read this!

The Definition of Ready (DoR) can be found at https://developers.minds.com/docs/handbook/how-we-work/#definition-of-ready-dor

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "Type::Feature" label and verify the feature you're about to submit isn't a duplicate.
--->

## Goal

<!---
Clearly outline the goal of the issue and why we are doing it
--->

In order to support many Minds posts and channels being available on Nostr, we need to store a reverse index of the nostr has to the entity. 

## What needs to be done

<!---
Clearly detail what needs to be done to achieve the goal set out above
--->

Subscribe to the new 'entities-ops' event stream and record relationship of its nostr id (hash) to entity urn.

### Why?

Nostr clients make a request for events based off a sha-256 hash of an event id. We don't have any mechanism of translating these hashes into entities in order to return a response. 

## QA

<!---
Consideration is given to how the issue will be demoed and tested prior to being deployed
--->

Use the provided api endpoint to return an entity from a nostr hash

## UX/Design

<!---
Sufficient wireframes have been presented and understood by the development team. Full mockups are not required and re-usable components are encouraged.
--->

N/A

## Personas

<!---
Who does this user impact and why (optional)
--->

Digital Nomads

## Experiments

<!---
Experiments are not required, but it should be made clear if one is expected. If experiments are required, the hypothesis should be outlined.
--->

N/A


## Acceptance Criteria

<!--
Everyone should be able to understand what is expected to be delivered
--->

- [ ] A new table that maps nostr hashes to entity urns
- [ ] A mechanism (preferably a background runner) that does the reverse indexing stage
- [ ] An api endpoint that will return an entity from a nostr hash
- [ ] **Nostr relay and communication with Nostr is out of scope for this task and is not required or expected**
- [ ] Spec tests

## Definition of Ready Checklist

- [x] Definition Of Done (DoD)
- [x] Acceptance criteria
- [x] Weighted
- [x] QA
- [x] UX/Design
- [x] Personas
- [x] Experiments

Story points: 5.0
Video unmute metric misfiring

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "T - Bug" label and verify the issue you're about to submit isn't a duplicate.
--->

### Summary

Reported by Mani

The video unmute metric is being misfired when scrolling a video in and out of the viewport.

### Steps to reproduce

1. Open Snowplow Debugger.
2. Make sure autoplay is turned on.
3. Scroll down to a video (go to a channel video feed maybe)
4. Scroll up so the video goes out of the viewport
5. Scroll back down - the video restarts (potentially its own bug) and an event is fired for unmute. 

### Platform information 

Web

### What is the current *bug* behavior?

Event misfire

### What is the expected *correct* behavior?

Should only fire on unmute events.

### Relevant logs and/or screenshots

https://www.loom.com/share/65cd90ed776f4f9eb89b5e16543a6461 

### Possible fixes

https://gitlab.com/minds/front/blob/cd7a7412c4159dc3556070bead00b7186fe74e13/src/app/modules/media/components/video-player/player.component.ts#L336

Story points: 0.0
Implement Prometheus Metrics for Testing Connectivity to External Dependencies

<!---
Please read this!

The Definition of Ready (DoR) can be found at https://developers.minds.com/docs/handbook/how-we-work/#definition-of-ready-dor

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "Type::Feature" label and verify the feature you're about to submit isn't a duplicate.
--->

## Goal

<!---
Clearly outline the goal of the issue and why we are doing it
--->

Currently for our readiness and liveness probes we check the `/api/v1/minds/config` endpoint, which does not evaluate connectivity to external dependencies. We can implement a dedicated health endpoint that tries to connect to these external services and returns a `200` status code only if the engine is able to communicate with them.

## What needs to be done

<!---
Clearly detail what needs to be done to achieve the goal set out above
--->

Implement a health endpoint that confirms connectivity to the following services:

- Cassandra
- Elasticsearch
- Redis
- Sendgrid
- SQS
- Permaweb
- Web3 Server

We'll also want to update the readiness and liveness probes in our Helm chart.

## QA

<!---
Consideration is given to how the issue will be demoed and tested prior to being deployed
--->

We can test this in Sandbox by applying a NetworkPolicy that restricts access to the required dependencies, which should cause the endpoint to fail and therefore the pod to show as unready.

## UX/Design

<!---
Sufficient wireframes have been presented and understood by the development team. Full mockups are not required and re-usable components are encouraged.
--->

N/A

## Personas

<!---
Who does this user impact and why (optional)
--->

N/A

## Experiments

<!---
Experiments are not required, but it should be made clear if one is expected. If experiments are required, the hypothesis should be outlined.
--->

N/A

## Acceptance Criteria

<!--
Everyone should be able to understand what is expected to be delivered
--->

- [ ] Health endpoint is added to engine
- [ ] Helm charted is updated with readiness and liveness probes

## Definition of Ready Checklist

- [X] Definition Of Done (DoD)
- [X] Acceptance criteria
- [ ] Weighted
- [X] QA
- [X] UX/Design
- [X] Personas
- [X] Experiments

Story points: 3.0
Move Sockets JWT Secret to Vault

<!---
Please read this!

The Definition of Ready (DoR) can be found at https://developers.minds.com/docs/handbook/how-we-work/#definition-of-ready-dor

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "Type::Feature" label and verify the feature you're about to submit isn't a duplicate.
--->

## Goal

<!---
Clearly outline the goal of the issue and why we are doing it
--->

We need to move from a hard-coded JWT secret for Sockets into Vault so that we don't have to copy a file on new Terraform deployments.

## What needs to be done

<!---
Clearly detail what needs to be done to achieve the goal set out above
--->

TODO

## QA

<!---
Consideration is given to how the issue will be demoed and tested prior to being deployed
--->

TODO

## UX/Design

<!---
Sufficient wireframes have been presented and understood by the development team. Full mockups are not required and re-usable components are encouraged.
--->

N/A

## Personas

<!---
Who does this user impact and why (optional)
--->

N/A

## Experiments

<!---
Experiments are not required, but it should be made clear if one is expected. If experiments are required, the hypothesis should be outlined.
--->

N/A

## Acceptance Criteria

<!--
Everyone should be able to understand what is expected to be delivered
--->

- [ ] Sockets JWT Secret is being pulled from Vault
- [ ] Able to set socket variables via env variables https://gitlab.com/minds/engine/-/blob/master/settings.example.php#L256

## Definition of Ready Checklist

- [ ] Definition Of Done (DoD)
- [ ] Acceptance criteria
- [ ] Weighted
- [ ] QA
- [ ] UX/Design
- [ ] Personas
- [ ] Experiments

M

Story points: 2.0
API to trigger nostr sync

<!---
Please read this!

The Definition of Ready (DoR) can be found at https://developers.minds.com/docs/handbook/how-we-work/#definition-of-ready-dor

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "Type::Feature" label and verify the feature you're about to submit isn't a duplicate.
--->

## Goal

<!---
Clearly outline the goal of the issue and why we are doing it
--->


## What needs to be done

<!---
Clearly detail what needs to be done to achieve the goal set out above
--->


## QA

<!---
Consideration is given to how the issue will be demoed and tested prior to being deployed
--->


## UX/Design

<!---
Sufficient wireframes have been presented and understood by the development team. Full mockups are not required and re-usable components are encouraged.
--->


## Personas

<!---
Who does this user impact and why (optional)
--->


## Experiments

<!---
Experiments are not required, but it should be made clear if one is expected. If experiments are required, the hypothesis should be outlined.
--->


## Acceptance Criteria

<!--
Everyone should be able to understand what is expected to be delivered
--->

- [ ] ...

## Definition of Ready Checklist

- [ ] Definition Of Done (DoD)
- [ ] Acceptance criteria
- [ ] Weighted
- [ ] QA
- [ ] UX/Design
- [ ] Personas
- [ ] Experiments

Story points: 0.0
Blogs dropdown UI issues

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "T - Bug" label and verify the issue you're about to submit isn't a duplicate.
--->

### Summary

Issues with the dropdown in blogs. 

### Steps to reproduce

Run blogs Cypress test against local or sandbox.

Replicate outside of Cypress

1. In firefox, create a new blog
2. Set a banner and a long title
3. Make your window small like the screenshot
4. Open the license menu

### Platform information 

Cypress against web.

### What is the current *bug* behavior?

Menu z-index messed up

### What is the expected *correct* behavior?

Menu should be correctly displayed. Test should green light. (may be other issues)

### Relevant logs and/or screenshots

![image](/uploads/fa173bc0f9b5378a0ebd0cf4c90fea4e/image.png)

### Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

Story points: 2.0
Create an 'entities-ops' EventStream

<!---
Please read this!

The Definition of Ready (DoR) can be found at https://developers.minds.com/docs/handbook/how-we-work/#definition-of-ready-dor

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "Type::Feature" label and verify the feature you're about to submit isn't a duplicate.
--->

## Goal

<!---
Clearly outline the goal of the issue and why we are doing it
--->

Create a new EventStream for changes to entities. For example: create, update, delete.

## What needs to be done

<!---
Clearly detail what needs to be done to achieve the goal set out above
--->

Whenever an entity is created, updated or deleted, submit an event to an EventStream that we can later subscribe to. The data team require this for their cassandra extra. We can use this in place of our SearchIndexDispatcher to better replicate cassandra -> elasticsearch.

### Event fields

| field | type | example |
| ------ | ------ | ------ |
| type | enum/string| create,update,delete |
| entity_urn | string | urn:user:123 |

### Complications

Previously we had an issue on signup when two pulsar events tried to be published. The first event succeeded but then 
closed the publisher connection, therefore the second event failed throwing an exception. A small refactor **may** be needed to create a **shared** connection pool, that should only be destroyed on process termination.

## QA

<!---
Consideration is given to how the issue will be demoed and tested prior to being deployed
--->

Ensure create, update and delete actions happen without errors. 

## UX/Design

<!---
Sufficient wireframes have been presented and understood by the development team. Full mockups are not required and re-usable components are encouraged.
--->

N/A

## Personas

<!---
Who does this user impact and why (optional)
--->

Data science

## Experiments

<!---
Experiments are not required, but it should be made clear if one is expected. If experiments are required, the hypothesis should be outlined.
--->

N/A


## Acceptance Criteria

<!--
Everyone should be able to understand what is expected to be delivered
--->

- [x] A new event stream
- [ ] Possible refactor the connection termination? See complications above. 
- [x] An example subscription showing the stream working
- [ ] Documentation
- [x] Spec tests

## Definition of Ready Checklist

- [ ] Definition Of Done (DoD)
- [ ] Acceptance criteria
- [ ] Weighted
- [ ] QA
- [ ] UX/Design
- [ ] Personas
- [ ] Experiments

Story points: 5.0