Package: abnosql
NoSQL Abstraction LibraryBasic CRUD and query support for NoSQL databases, allowing for portable cloud native applicationsAWS DynamoDBAzure Cosmos NoSQLThis library is not intended to create databases/tables, use Terraform/ARM/CloudFormation etc for thatWhy not just use the name 'nosql' or 'pynosql'? because they already exist on pypi :-)NoSQL Abstraction LibraryInstallationUsageAPI DocsQueryingIndexesUpdatesExistence CheckingSchema ValidationPartition KeysPaginationAuditChange Feed / Stream SupportClient Side EncryptionConfigurationAWS DynamoDBAzure Cosmos NoSQLPlugins and HooksTestingAWS DynamoDBAzure Cosmos NoSQLCLIFuture Enhancements / IdeasInstallationpip install 'abnosql[dynamodb]'
pip install 'abnosql[cosmos]'For optionalclient sidefield level envelope encryptionpip install 'abnosql[aws-kms]'
pip install 'abnosql[azure-kms]'By default, abnosql does not include database dependencies.  This is to facilitate packaging
abnosql into AWS Lambda or Azure Functions (for example), without over-bloating the packagesUsagefrom abnosql import table
import os

os.environ['ABNOSQL_DB'] = 'dynamodb'
os.environ['ABNOSQL_KEY_ATTRS'] = 'hk,rk'

item = {
    'hk': '1',
    'rk': 'a',
    'num': 5,
    'obj': {
        'foo': 'bar',
        'num': 5,
        'list': [1, 2, 3],
    },
    'list': [1, 2, 3],
    'str': 'str'
}

tb = table('mytable')

# create/replace
tb.put_item(item)

# update - using ABNOSQL_KEY_ATTRS
updated_item = tb.put_item(
    {'hk': '1', 'rk': 'a', 'str': 'STR'},
    update=True
)
assert updated_item['str'] == 'STR'

# bulk
tb.put_items([item])

# note partition/hash key should be first kwarg
assert tb.get_item(hk='1', rk='a') == item

assert tb.query({'hk': '1'})['items'] == [item]

# scan
assert tb.query()['items'] == [item]

# be careful not to use cloud specific statements!
assert tb.query_sql(
    'SELECT * FROM mytable WHERE mytable.hk = @hk AND mytable.num > @num',
    {'@hk': '1', '@num': 4}
)['items'] == [item]

tb.delete_item({'hk': '1', 'rk': 'a'})API DocsSeeAPI DocsQueryingquery()performs DynamoDBQueryusing KeyConditionExpression (ifkeysupplied) and exact match on FilterExpression if filters are supplied.  For Cosmos, SQL is generated.  This is the safest/most cloud agnostic way to query and probably OK for most use cases.query_sql()performs DynamodbExecuteStatementpassing in the suppliedPartiQLstatement.  Cosmos uses the NoSQLSELECTsyntax.During mocked tests,SQLGlotis used toexecutethe statement, so results may differ...Care should be taken withquery_sql()to not to use SQL features that are specific to any specific provider (breaking the abstraction capability of using abnosql in the first place)IndexesBeyond partition and range keys defined on the table, indexes currently have limited support within abnosqlThe DynamoDB implemention ofquery()allows asecondary indexto be specified via optionalindexkwargCosmoshas Range, Spatial and Composite indexes, however the abnosql library does not do anything yet withindexkwarg inquery()implementation.Updatesput_item()andput_items()supportupdateboolean attribute, which if supplied will do anupdate_item()on DynamoDB, and apatch_item()on Cosmos.  For this to work however, you must specify the key attribute names, either viaABNOSQL_KEY_ATTRSenv var as a comma separated list (eg perhaps multiple tables all share common partition/range key scheme), or as thekey_attrsconfig item  when instantiating the table, eg:tb = table('mytable', {'key_attrs': ['hk', 'rk']})If you don't need to do any updates and only need to do create/replace, then these key attribute names do not need to be suppliedAll items being updated must actually exist first, or else exception raisedExistence CheckingIfcheck_existsconfig attribute isTrue, then CRUD operations will raise exceptions as follows:get_item()raisesNotFoundExceptionif item doesnt existput_item()raisesExistsExceptionif item already existsput_item(update=True)raisesNotFoundExceptionif item doesnt exist to updatedelete_item()raisesNotFoundExceptionif item doesnt existThis adds some delay overhead as abnosql must check if item existsThis can also be enabled by setting environment variableABNOSQL_CHECK_EXISTS=TRUEIf for some reason you need to override this behaviour once enabled forput_item()create operation,
you can passabnosql_check_exists=Falseinto the item (this gets popped out so not persisten), which
will allow create operation to overwrite the existing item without throwingExistsExceptionSchema Validationconfigcan define jsonschema to validate upon create or update operations (viaput_item())Combination of the following config attributes supportedschema: jsonschema dict or yaml string, applied to both create and updatecreate_schema: jsonschema dict/yaml only on createupdate_schema: jsonschema dict/yaml only on updateschema_errmsg: override default error message on both create and updatecreate_schema_errmsg: override default error message on createupdate_schema_errmsg: override default error message on updateYou can get details of validation errors throughe.to_problem()ore.detailNOTE:key_attrsrequired when updating (seeUpdates)Partition KeysA few methods such asget_item(),delete_item()andquery()need to know partition/hash keys as defined on the table.  To avoid having to configure this or lookup from the provider, the convention used is that the first kwarg or dictionary item is the partition key, and if supplied the 2nd is the range/sort key.Paginationqueryandquery_sqlacceptlimitandnextoptional kwargs and returnnextin response. Use these to paginate.This works for AWS DyanmoDB, however Azure Cosmos has a limitation with continuation token for cross partitions queries (seePython SDK documentation).  For Cosmos, abnosql appends OFFSET and LIMIT in the SQL statement if not already present, and returnsnext.limitis defaulted to 100.  See the tests for examplesAuditput_item()andput_items()take an optionalaudit_userkwarg.  If supplied, absnosql will add the following to the item:createdBy- value ofaudit_user, added if does not exist in item supplied to put_item()createdDate- UTC ISO timestamp string, added if does not existmodifiedBy- value ofaudit_useralways addedmodifiedDate- UTC ISO timestamp string, always addedYou can also specifyaudit_useras config attribute to table.  If you prefer snake_case over CamelCase, you can set env varABNOSQL_CAMELCASE=FALSENOTE: created* will only be added ifupdateis not True in aput_item()operationChange Feed / Stream SupportAWS DynamoDBStreamsallow Lambda functions to be triggered upon create, update and delete table operations.  The event sent to the lambda (seeaws docs) containseventNameandeventSourceARN, where:eventName- name of event, egINSERT,MODIFYorREMOVE(seehere)eventSourceARN- ARN of the table nameThis allows a single stream processor lambda to process events from multiple tables (eg for writing into ElasticSearch)Like DynamoDB,Azure CosmosDBsupportschange feeds, however the event sent to the function (currently) omits the event source (table name) and only delete event names are available if apreview change feed modeis enabled, which needs explicit enablement for.Because both the eventName and eventSource are ideally needed (irrespective of preview mode or not), abnosql library automatically adds thechangeMetadatato an item during create, update and delete, eg:item = {
    "hk": "1",
    "rk": "a",
    "changeMetadata": {
        "eventName": "INSERT",
        "eventSource": "sometable"
    }
}Because no REMOVE event is sent at all without preview change feed mode above - abnosql must first update the item, and then delete it.  This is also needed for the eventSource / table name to be captured in the event, so unfortunately until Cosmos supports both attributes, update is needed before a delete.  5 second synchronous sleep is added by default between update and delete to allow CosmosDB to send the update event (0 seconds results in no update event).  This can be controlled withABNOSQL_COSMOS_CHANGE_META_SLEEPSECSenv var (defaults to5seconds), and disabled by setting to0This behaviour is enabled by default, however can be disabled by settingABNOSQL_COSMOS_CHANGE_METAenv var toFALSEorcosmos_change_meta=Falsein table config.ABNOSQL_CAMELCASE=FALSEenv var can also be used to change attribute names used to snake_case if neededTo write an Azure Function / AWS Lambda that is able to process both DynamoDB and Cosmos events, look forchangeMetadatafirst and if present use that otherwise look foreventNameandeventSourceARNin the event payload assuming its DynamoDBClient Side EncryptionIf configured in table config withkmsattribute, abnosql will perform client side encryption using AWS KMS or Azure KeyVaultEach attribute value defined in the config is encrypted with a 256-bit AES-GCM data key generated for each attribute value:awsusesAWS Encryption SDK for Pythonazureusespython cryptographyto generate AES-GCM data key, encrypt the attribute value and then uses an RSA CMK in Azure Keyvault to wrap/unwrap (envelope encryption) the AES-GCM data key.  The module uses theazure-keyvaults-keyspython SDK for wrap/unrap functionality of the generated data key (Azure doesnt support generate data key as AWS does)Both providers use a256-bit AES-GCMgenerated data key with AAD/encryption context (Azure provider uses a 96-nonce).  AES-GCM is an Authenticated symmetric encryption scheme used by both AWS and Azure (andHashicorp Vault)See alsoAWS Encryption Best PracticesExample config:{
    'kms': {
        'key_ids': ['https://foo.vault.azure.net/keys/bar/45e36a1024a04062bd489db0d9004d09'],
        'key_attrs': ['hk', 'rk'],
        'attrs': ['obj', 'str']
    }
}Where:key_ids: list of AWS KMS Key ARNs or Azure KeyVault identifier (URL to RSA CMK).  This is picked up viaABNOSQL_KMS_KEYSenv var as a comma separated list (NOTE: env var recommended to avoid provider specific code)key_attrs: list of key attributes in the item from which the AAD/encryption context is set.  Taken fromABNOSQL_KEY_ATTRSenv var or tablekey_attrsif defined thereattrs: list of attributes keys to encryptkey_bytes: optional for azure, use your own AESGCM key if specified, otherwise generate oneIfkmsconfig attribute is present, abnosql will look for theABNOSQL_KMSprovider to load the appropriate provider KMS module (eg "aws" or "azure"), and if not present use default depending on the database (eg cosmos will use azure, dynamodb will use aws)In example above, the key_attrs['hk', 'rk']are used to define the encryption context / AAD used, and attrs['obj', 'str']what attributes to encrypt/decryptWith an item:{
    'hk': '1',
    'rk': 'b',
    'obj': {'foo':'bar'},
    'str': 'foobar'
}The encryption context / AAD is set to hk=1 and rk=b and obj and str values are encryptedIf you don't want to use any of these providers, then you can useput_item_preandget_item_posthooks to perform your own client side encryptionSee alsoAWS Multi-region encryption keysand setABNOSQL_KMS_KEYSenv var as comma list of ARNsConfigurationIt is recommended to use environment variables where possible to avoid provider specific application codeifABNOSQL_DBenv var is not set, abnosql will attempt to apply defaults based on available environment variables:AWS_DEFAULT_REGION- sets database todynamodb(seeaws docs)FUNCTIONS_WORKER_RUNTIME- sets database tocosmos(seeazure docs)AWS DynamoDBSet the following environment variable and use the usual AWS environment variables that boto3 usesABNOSQL_DB= "dynamodb"Or set the boto3 session in the configfrom abnosql import table
import boto3

tb = table(
    'mytable',
    config={'session': boto3.Session()},
    database='dynamodb'
)Azure Cosmos NoSQLSet the following environment variables:ABNOSQL_DB= "cosmos"ABNOSQL_COSMOS_ACCOUNT= your database accountABNOSQL_COSMOS_ENDPOINT= drived fromABNOSQL_COSMOS_ACCOUNTif not setABNOSQL_COSMOS_CREDENTIAL= your cosmos credential, useAzure Key Vault Referencesif using Azure Functions.  Don't set to use DefaultAzureCredential / managed identity.ABNOSQL_COSMOS_DATABASE= cosmos databaseOR- use the connection string format:ABNOSQL_DB= "cosmos://account@credential:database" or "cosmos://account@:database" to use managed identity (credential could also be "DefaultAzureCredential")Alternatively, define in config (though ideally you want to use env vars to avoid application / environment specific code).from abnosql import table

tb = table(
    'mytable',
    config={'account': 'foo', 'database': 'bar'},
    database='cosmos'
)Plugins and Hooksabnosql uses pluggy and registers in theabnosql.tablenamespaceThe following hooks are availableset_config- set configget_item_post- called afterget_item(), can return modified dataput_item_preput_item_postput_items_postdelete_item_postSee theTableSpecsand exampletest_hooks()TestingAWS DynamoDBUsemotopackage andabnosql.mocks.mock_dynamodbxmock_dynamodbx is used for query_sql and only needed if/until moto provides full partiql supportExample:from abnosql.mocks import mock_dynamodbx 
from moto import mock_dynamodb

@mock_dynamodb
@mock_dynamodbx  # needed for query_sql only
def test_something():
    ...More examples intests/test_dynamodb.pyAzure Cosmos NoSQLUserequestspackage andabnosql.mocks.mock_cosmosExample:from abnosql.mocks import mock_cosmos
import requests

@mock_cosmos
@responses.activate
def test_something():
    ...More examples intests/test_cosmos.pyCLISmall abnosql CLI installed with few of the commands aboveUsage: abnosql [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  delete-item
  get-item
  put-item
  put-items
  query
  query-sqlTo install dependenciespip install 'abnosql[cli]'Example querying table in Azure Cosmos, with cosmos.json config file containing endpoint, credential and database$ abnosql query-sql mytable 'SELECT * FROM mytable' -d cosmos -c cosmos.json
partkey      id      num  obj                                          list       str
-----------  ----  -----  -------------------------------------------  ---------  -----
p1           p1.1      5  {'foo': 'bar', 'num': 5, 'list': [1, 2, 3]}  [1, 2, 3]  str
p2           p2.1      5  {'foo': 'bar', 'num': 5, 'list': [1, 2, 3]}  [1, 2, 3]  str
p2           p2.2      5  {'foo': 'bar', 'num': 5, 'list': [1, 2, 3]}  [1, 2, 3]  strFuture Enhancements / Ideasclient side encryptiontest pagination & exception handlingGoogle Firestoresupport, ideally in the core library (though could be added outside via use of the plugin system).  Would need something likeFireSQLimplemented for oython, maybe via sqlglotGoogle VaultKMS supportHashicorp VaultKMS supportSimple caching (maybe) using globals (used for AWS Lambda / Azure Functions)PostgresSQL support using JSONB column (seeherefor example).  Would be nice to avoid an ORM and having to define a model for each table...blob storage backend? could use something similar toNoDBbut maybe combined withsmart_openand DuckDB'sHive PartitioningRedis..Hook implementations to write to ElasticSearch / OpenSearch for better searching.  Useful when not able to useAWS Stream ProcessorsAzure Change Feed, orElasticstore. Why? because not all databases support stream processing, and if they do you don't want the hastle of usingCDC