Package: ocean-aquarius
Aquarius🐋 Aquarius provides an off-chain database cache for metadata that is published on-chain.
This enables faster query operations on datasets metadata.
The latest version of Aquarius consist of a Flask application to support fetching and
searching metadata and a blockchain events monitor that picks up new metadata published
on-chain and stores it in the database backend (elasticsearch).It's part of theOcean Protocolsoftware stack.🐲🦑 THERE BE DRAGONS AND SQUIDS. This is in alpha state and you can expect running into problems. If you run into them, please open upa new issue. 🦑🐲What Aquarius doesAquarius runs a Flask RESTful server to support fetching and searching metadata of datasets that are published on-chainThe metadata is published on-chain via the Metadata smartcontract:Metadata is first compressed then published on-chainThe compressed metadata on-chain is not kept in storage, but rather is captured in an event log namedMetadataCreatedThe id (DID) of the dataset asset is the Datatoken address, off-chain the did consist ofdid:op:prepended to the datatoken addressAquarius runs an events monitor that watches the:MetadataCreatedevent from theMetadatasmartcontractReads the eventsdataargument, decompresses the metadata json object
then runs schema validation before saving it to the databaseLOG_JOIN,LOG_EXITandLOG_SWAPevents from theBPoolsmartcontractsAny of these events is an indication that liquidity and price have changedThe watcher reads the liquidity of each token in the pool and updates the
corresponding metadata in the cache. This information is added to the metadata
to allow sorting and searching by price and/or liquidity volumeSetupThe following environment variables are required for running Aquarius:# URL of ethereum network.
# Recommendation: when connecting to an official network, create an Infura project id and set this
# to use the Infura url including the project id
EVENTS_RPC
  examples:
  "http://172.15.0.3:8545", "wss://rinkeby.infura.io/ws/v3/INFURA_ID" 

# Use this to run the EventsMonitor in a thread from the main Flask app
EVENTS_ALLOW
  accepted values: 
    "0" to disable
    "1" to enable 

# Run the EventsMonitor in a separate process, overrides `EVENTS_ALLOW`.
# This is only used when running in `docker` container 
RUN_EVENTS_MONITOR
  accepted values: 
    "0" to disable
    "1" to enableAnd these are optional# Use this to decrypt metadata when read from the blockchain event logEVENTS_ECIES_PRIVATE_KEY# Path to abi files of the ocean contractsARTIFACTS_PATH# Path to the `address.json` file or any json file that has the deployed contracts addressesADDRESS_FILE# Specify the network name to use for reading the contracts addresses from the `ADDRESS_FILE`.# If not set, the netwrok name is derived from current network id or from the `EVENTS_RPC` valueNETWORK_NAME# Skip caching metadata of publishers that are not in this listALLOWED_PUBLISHERS# The blockNumber of `BFactory` deploymentBFACTORY_BLOCK# The blockNumber of `Metadata` contract deploymentMETADATA_CONTRACT_BLOCKFor Aquarius OperatorsIf you're developing a marketplace, you'll want to run Aquarius and several other components locally,
and the easiest way to do that is to use Barge. See the instructions
inthe Barge repository.For Aquarius API UsersHereis API documentation.If you have Aquarius running locally, you can find API documentation athttp://localhost:5000/api/v1/docsor maybehttp://0.0.0.0:5000/api/v1/docs.Tip 1: If that doesn't work, then tryhttps.Tip 2: If your browser shows the Swagger header across the top but says "Failed to load spec." then we found that, in Chrome, if we went tochrome://flags/#allow-insecure-localhostand toggled it to Enabled, then relaunched Chrome, it worked.More details about ontology of the metadata are atOEP-8.For Aquarius DevelopersGeneral Ocean Dev DocsOcean's Python code style and related "meta" developer docs are atoceanprotocol/dev-ocean repo.Running as a Docker containerFirst, clone this repository:gitclonegit@github.com:oceanprotocol/aquarius.gitcdaquarius/Then build the Docker imagedockerbuild-t"myaqua".Run Docker imagedockerrunmyaquaTo test with other ocean components inbargeset theAQUARIUS_VERSIONenvironment variable tomyaquaThen./start_ocean.shThe setup forAquariusand Alastic search inbargeis incompose-files/aquarius_elasticsearch.ymlRunning Locally, for Dev and TestFirst, clone this repository:gitclonegit@github.com:oceanprotocol/aquarius.gitcdaquarius/Then run elasticsearch database that is a requirement for Aquarius.exportES_VERSION=6.6.2exportES_DOWNLOAD_URL=https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-${ES_VERSION}.tar.gz
wget${ES_DOWNLOAD_URL}tar-xzfelasticsearch-${ES_VERSION}.tar.gz
./elasticsearch-${ES_VERSION}/bin/elasticsearch&Then install Aquarius's OS-level requirements:sudoaptupdate
sudoaptinstallpython3-devpython3.7-dev(Note: At the time of writing,python3-devwas for Python 3.6.python3.7-devis needed if you want to test against Python 3.7 locally.)Before installing Aquarius's Python package requirements, it's recommended to create and activate a virtualenv (or equivalent).At this point, an Elasticsearch database must already be running, now you can start the Aquarius server:pipinstall-rrequirements.txtexportFLASK_APP=aquarius/run.pyexportCONFIG_FILE=config.ini
flaskrunThat will use HTTP (i.e. not SSL/TLS).The proper way to run the Flask application is using an application server such as Gunicorn. This allow you to run using SSL/TLS.
You can generate some certificates for testing by doing:opensslreq-x509-newkeyrsa:4096-nodes-outcert.pem-keyoutkey.pem-days365and when it asks for the Common Name (CN), answerlocalhostThen edit the config fileconfig.iniso that:aquarius.url = http://localhost:5000Then execute this command:gunicorn--certfilecert.pem--keyfilekey.pem-b0.0.0.0:5000-w1aquarius.run:appConfigurationYou can pass the configuration using the CONFIG_FILE environment variable (recommended) or locating your configuration in config.ini file.In the configuration there are now two sections:oceandb: Contains different values to connect with oceandb. You can find more information about how to use OceanDBhere.resources: In this section we are showing the url in which the aquarius is going to be deployed.[resources]aquarius.url = http://localhost:5000TestingAutomatic tests are set up via Travis, executingtox.
Our tests use the pytest framework.New VersionThebumpversion.shscript helps bump the project version. You can execute the script using{major|minor|patch}as first argument, to bump the version accordingly.LicenseCopyright 2020 Ocean Protocol Foundation Ltd.Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.