Package: arango-crud
arango_crudThis respository wraps the basic CRUD operations on ArangoDB for practical use.
This is not an official library; the official python library ispyArango. The main reason to
use this over justrequestsis
authorization and server failure back-off. The main reason to use this over
pyArango is thread-safety, simpler interfaces, a more narrow focus, pep8
naming conventions, and complete support for either environment variable
configuration or code-as-configuration.The main reason to use pyArango over arango_crud is field validation and access
to AQL. If you want to use ArangoDB as a database, use pyArango or similar. If
you want to use ArangoDB as a disk-based cache, use arango_crud or similar.Note: This package recommends a time-to-live semantic. The TTL may be
set to "-1" in environment variables to be disabled, or "None" in code to be
disabled. A TTL index is only created when collections are initialized, so if
this library is used with TTL disabled and then TTL is enabled, one must
manually add the TTL indexes. Besides standard TTL usages, using a TTL means
that if there was a bug that leaked keys which was since patched, those keys
won't stay around forever. Furthermore, it means a small amount of key leakage,
such as through extremely unlikely race conditions which would be expensive
in either performance or developer time to fix, is not harmful to the long-term
health of the project.https://www.arangodb.com/arangodb-training-center/ttl-indexes/Note: This is not intended to provide much configurability for creating
databases or collections, providing only sane defaults for this particular
use-case. It's recommended to use a migration structure where databases and
collections are only initialized once and to call the appropriate HTTP endpoint
directly:https://www.arangodb.com/docs/stable/http/collection-creating.htmlUsageInstallationSupports python 3.7 or higher.pip install arango_crudInitializeCode-as-configuration BasicAuthfromarango_crudimport(Config,BasicAuth,RandomCluster,StepBackOffStrategy)config=Config(cluster=RandomCluster(urls=['http://127.0.0.1:8529']),# see Cluster Stylestimeout_seconds=3,back_off=StepBackOffStrategy([0.1,0.5,1,1,1]),# see Back Off Strategiesauth=BasicAuth(username='root',password=''),ttl_seconds=31622400)Code-as-configuration JWTfromarango_crudimport(Config,JWTAuth,JWTDiskCache,RandomCluster,StepBackOffStrategy)config=Config(cluster=RandomCluster(urls=['http://localhost:8529']),timeout_seconds=3,back_off=StepBackOffStrategy(steps=[0.1,0.5,1,1,1]),ttl_seconds=31622400,auth=JWTAuth(username='root',password='',cache=JWTDiskCache(# See JWT Cacheslock_file='.arango_jwt.lock',lock_time_seconds=10,store_file='.arango_jwt')))# encouraged for easier performance tracing, not required. happens on first# request otherwise. Fetches the JWT token if it does not exist.config.prepare()Environment variables BasicAuthtest.pyfromarango_crudimportenv_configconfig=env_config()config.prepare()# recommended, not requiredrun.sh#!/usr/bin/env bash# Cluster urls are separated by a commaexportARANGO_CLUSTER=http://localhost:8529exportARANGO_CLUSTER_STYLE=randomexportARANGO_TIMEOUT_SECONDS=3exportARANGO_BACK_OFF=stepexportARANGO_BACK_OFF_STEPS=0.1,0.5,1,1,1exportARANGO_TTL_SECONDS=31622400exportARANGO_AUTH=basicexportARANGO_AUTH_USERNAME=rootexportARANGO_AUTH_PASSWORD=pythontest.pyEnvironment variables JWTtest.pyfromarango_crudimportenv_configconfig=env_config()run.sh#!/usr/bin/env bash# Cluster urls are separated by a commaexportARANGO_CLUSTER=http://localhost:8529exportARANGO_CLUSTER_STYLE=randomexportARANGO_TIMEOUT_SECONDS=3exportARANGO_BACK_OFF=stepexportARANGO_BACK_OFF_STEPS=0.1,0.5,1,1,1exportARANGO_TTL_SECONDS=31622400exportARANGO_AUTH=jwtexportARANGO_AUTH_USERNAME=rootexportARANGO_AUTH_PASSWORD=exportARANGO_AUTH_CACHE=diskexportARANGO_AUTH_CACHE_LOCK_FILE=.arango_jwt.lockexportARANGO_AUTH_CACHE_LOCK_TIME_SECONDS=10exportARANGO_AUTH_CACHE_STORE_FILE=.arango_jwt
pythontest.pyCRUDTo make these runnable environment variables must be set and ArangoDB
needs to be reachable. Here are the configurations for ArangoDB running
locally on default development settings:Windows:SETARANGO_CLUSTER=http://localhost:8529SETARANGO_CLUSTER_STYLE=randomSETARANGO_TIMEOUT_SECONDS=3SETARANGO_BACK_OFF=stepSETARANGO_BACK_OFF_STEPS=0.1,0.5,1,1,1SETARANGO_TTL_SECONDS=31622400SETARANGO_AUTH=basicSETARANGO_AUTH_USERNAME=rootSETARANGO_AUTH_PASSWORD=*Nix:#!/usr/bin/env bashexportARANGO_CLUSTER=http://localhost:8529exportARANGO_CLUSTER_STYLE=randomexportARANGO_TIMEOUT_SECONDS=3exportARANGO_BACK_OFF=stepexportARANGO_BACK_OFF_STEPS=0.1,0.5,1,1,1exportARANGO_TTL_SECONDS=31622400exportARANGO_AUTH=basicexportARANGO_AUTH_USERNAME=rootexportARANGO_AUTH_PASSWORD=fromarango_crudimportenv_configimporttimeconfig=env_config()config.prepare()db=config.database('my_db')db.create_if_not_exists()coll=db.collection('users')coll.create_if_not_exists()# The simplest interfacecoll.create_or_overwrite_doc('tj',{'name':'TJ'})coll.read_doc('tj')# {'name': 'TJ'}coll.force_delete_doc('tj')# True# non-expiringcoll.create_or_overwrite_doc('tj',{'name':'TJ'},ttl=None)coll.force_delete_doc('tj')# custom expirations with touching. Note that touching a document is not# a supported atomic operation on ArangoDB and is hence faked with# read -> compare_and_swap. Presumably if the CAS fails the document was# touched recently anyway.coll.create_or_overwrite_doc('tj',{'name':'TJ'},ttl=30)# Nonecoll.touch_doc('tj',ttl=60)# Truecoll.force_delete_doc('tj')# True# Alternative interface. For anything except one-liners, usually nicer.doc=coll.document('tj')doc.body['name']='TJ'doc.create()# Truedoc.body['note']='Pretty cool'doc.compare_and_swap()# True# We may use etags to avoid redownloading an unchanged document, but be careful# if you are modifying the body.# Happy case:doc2=coll.document('tj')doc2.read()# loads {'name': 'TJ', 'note': 'Pretty cool'} from networkdoc.read_if_remote_newer()# 304 not modified, returns Falsedoc2.read_if_remote_newer()# 304 not modified, returns Falsedoc.body['note']='bar'doc.compare_and_swap()doc.read_if_remote_newer()# 304 not modified, returns Falsedoc2.read_if_remote_newer()# loads {'name': 'TJ', 'note': 'bar'} from network, returns True# Where it can get dangerousdoc.body['note']='foo'print(doc.body)# {'name': 'TJ', 'note': 'foo'}doc.read()# always a complete downloadprint(doc.body)# {'name': 'TJ', 'note': 'bar'}doc.read_if_remote_newer()# no changes on server since last read; 304 not modified, returns Falseprint(doc.body)# {'name': 'TJ', 'note': 'bar'}doc.body['note']='foo'print(doc.body)# {'name': 'TJ', 'note': 'foo'}doc.read_if_remote_newer()# no changes on server since last read; 304 not modified, returns Falseprint(doc.body)# {'name': 'TJ', 'note': 'foo'}doc.read()print(doc.body)# {'name': 'TJ', 'note': 'bar'}doc.compare_and_delete()# True# Simple cachingforiinrange(2):doc=coll.document('tj')hit=doc.read()ifhit:doc.compare_and_swap()# refreshes TTL, usefulness dependselse:# .... expensive computation ....doc.body={'name':'TJ','note':'Pretty cool'}doc.create_or_overwrite()print(f'cached value (loop{i+1}/2) (hit:{hit}):{doc.body}')The following is in a separate code-block and is commented out to prevent
accidentally copy+paste into somewhere it should not be pasted. When running
tests it's helpful to cleanup the collections and databases afterward. It's
encouraged that if you do not need to delete collections and databases on
production these operations are disabled to help prevent developer error, which
is done by settingARANGO_DISABLE_DATABASE_DELETEandARANGO_DISABLE_COLLECTION_DELETEtotrueThese environment variables are
treated astrueunless explicitly set tofalse. This is not a substitute
for good backups and should not be considered a security feature.# coll.force_delete()# db.force_delete()ContributingThis package adheres to pep8 guidelines unless an exception is listed in.flake8. Comments are explicitly line-broken at 80 characters. Code
complexity measures (AbcComplexity, etc) are not used. This measures code
coverage and a build of below 70% code coverage is considered failing. Note
that the word "unit test" is avoided - if it's possible to test a line of
code without mocking or accessing private variables that is preferred. PRs
which reduce code coverage must include an explanation of why.The examples directory should not contain non-functional lines of code, so
instead ofbar=foo()ifbarisNone:print('Foo gave none!')# prints Fooelse:print('Something went wrong!')it should be the easier to read assert variant, which plays friendlier with
automated testing that the examples actually work:bar=foo()assertbarisNoneHence any PR where the coverage in the examples directory is less than 98%
when runningcoverage run --rcfile=.coveragerc_examples examples/run_all.pywill have changes requested. The lines which do not run should be only due to
random chance.This repository is focused specifically on using ArangoDB as a disk-based
cache. Functionality which doesn't support that use-case will have their PR
closed with the recommendation that they fork. So AQL or graph support would
likely be closed, but (bulk) get/set operations or concurrency-safe patches
will likely be merged.Inheritance is to be avoided, preferring delegation which respects contracts.
Interfaces are not included in this, where an interface is a class where all
the functions simply raiseNotImplementedErrorand there is no constructor.Setup Development (Windows)Install ArangoDBon default
development settings.python -m venv venv
python -m pip install --upgrade pip"venv/Scripts/activate.bat"python -m pip install -r dev_requirements.txt"scripts/windows_dev_env.bat"coverage run -m unittest discover -s tests
coverage combine
coverage report
coverage run --rcfile=.coveragerc_examples examples/run_all.py
coverage reportSetup Development (*Nix)dockerpullarangodb/arangodb
dockerrun-eARANGO_NO_AUTH=1-p8529/tcparangodb/arangodbarangod--server-endpointtcp://0.0.0.0:8529
python-mvenvvenv
python-mpipinstall--upgradepip
.venv/bin/activate
.scripts/nix_dev_env.sh
python-mpipinstall-rdev_requirements.txt# This pulls from .coveragerc to handle multiprocessingcoveragerun-munittestdiscover-stests
coveragecombine
coveragereport
coveragerun--rcfile=.coveragerc_examplesexamples/run_all.py
coveragereportCluster StylesWhen working with an ArangoDB cluster, it's important that the clients
distribute their requests amongst the various coordinators. The request
styles supported arerandomandweighted-random. Round-robin and
similar are avoided as they cannot be made thread-safe and performant
without context.RandomA random url is selected from the cluster for each request with equal
probability among all urls.Weighted RandomA random node in the cluster is selected on each request, except there may be
a different probability for different urls. This is useful if, for example,
one of the coordinators is running on a larger server than the rest.Example:fromarango_crudimportWeightedRandomClustercluster=WeightedRandomCluster(urls=['http://localhost:8529','http://localhost:8530','http://localhost:8531'],weights=[1,2,1])This will select port 8529 1/4 of the time, 8530 1/2 of the time, and 8531 1/4
of the time. If one prefers to set the exact percentages just ensure the
weights sum to one (i.e.,0.25, 0.5, 0.25)Example environment variables:#!/usr/bin/env bashexportARANGO_CLUSTER=http://localhost:8529,http://localhost:8530,http://localhost:8531exportARANGO_CLUSTER_STYLE=weighted-randomexportARANGO_CLUSTER_WEIGHTS=1,2,1Alternatives to Environment VariablesAlthough environment variables are sometimes extremely convenient, they can
also be painful in other development environments. One can painlessly switch
these out for their preferred storage mechanism sinceenv_configaccepts
a dictionary which it uses to load variables from. Note thatenv_configwill exclusively use that dictionary - it will not fall back and use an
environment variable if something is missing.The only caveat is that for simplicity of development and to reuse the same
documentation, the keys need to be screaming snake case and it will not
make use of nesting. If one prefers they can massage the data into this format
after loading to get more conventional looking configuration files. One can
also simply massage the data into the arguments forConfigdirectly.arango_config.json{"ARANGO_CLUSTER":"http://localhost:8529,http://localhost:8530,http://localhost:8531","ARANGO_CLUSTER_STYLE":"weighted-random","__comment":"... see src/arango_crud/env_config.py for complete argument docs ..."}Which allows loading as follows:fromarango_crudimportenv_configimportjsonwithopen('arango_config.json')asfin:cfg=json.load(fin)arango_config=env_config(cfg)Server FailuresWhen a request fails due to a server-side issue it's usually desirable to try
again on a new coordinator. A small sleep is also helpful to avoid suddenly
massively spiking traffic to the coordinators whenever they hiccup. This
supports only astep-back-offpolicy. If the steps are[0.1, 0.5, 1]then
on the first server error this waits 0.1 seconds then tries again. If that
also fails this waits 0.5 seconds then tries again. If that fails this waits
1 second then tries again. If that fails, an error is raised.JWT Locking and StoreIt's usually not a good idea to create a lot of new tokens when a client is
misbehaving, as token generation is generally meant to be expensive in order to
be secure. Hence JWT is necessarily stateful on the Config - rather than just
being able to create network requests we first need to fetch the JWT.
Furthermore, we may need to refresh the token on arbitrary requests.The recommended way to handle JWT's cache isJWTDiskCache. A file will contain
the JWT and some metadata about it, which will be accessed in a safe way for
even highly concurrent environments, meaning that every instance running
arango_crud on the same machine using the same config will share JWT tokens
and will only create/renew the token once per renewal period. This overhead is
extremely minor for non-concurrent environments.If you're very confident that JWT generation is not going to be a significant
source of load and there is no multithreading, a naive approach can be enabled
with the cache styleNone. See the examples jwt_disk_example.py and
jwt_none_example.py