Package: gateaux
gateauxData structures and typing for FoundationDB."FoundationDB is a distributed database designed to handle large volumes of structured
data across clusters of commodity servers. It organizes data as an ordered key-value
store and employs ACID transactions for all operations." - taken fromhttps://github.com/apple/foundationdb/FoundationDB has, by design, a bare minimum of features. It presents an interface to
applications which reads and writes binary data with a few basic helper layers.
FoundationDB has no native support for rich data types (for exampledatetimeobjects)
nor provides any extended data validation. FoundationDB is designed to have layers of
abstraction built on top of it to provide additional features.The premise ofgateauxis that where you currently have, infdblibrary terms,tr[(some, data)] = (other, arbitrary, data)of unstructured data is to enforce strict
standardisation of data in these tuples while allowing more complex types (datetime,
ipaddress etc.). In addition, the concept of structures allows for easier developer
comprehension of what data is being stored in what FoundationDB keyspace.gateauxhas
a lot of code for not much of an interface, but it is designed to enforce structure and
types and therefore is over-engineered and over-tested by design so you don't have to
worry about your data structures in your upstream applications which usegateaux.If you usegateauxyour Python FoundationDB client code should look mostly the same,
you just can't mistakenly pack the wrong type or make mistakes before writing data. Such
additional validation is useful for larger codebases where you may be storing hundreds
ofkey=valueformats into FoundationDB and keeping track of them can be challenging.gateauxis a pure Python 3 (>=3.6) library which provides rich data type handling and
validation on top of the usualpack()andunpack()methods and extends thefdb.tuplebuilt-in layer. It is loosely modelled from the interfaces to relational
database object-relational mapper (RDBMS ORM) systems. Eachgateauxstructure
(comparable to a "model" in the ORM world) effectively formats one single key/value pair
at a time with more rigid validation than thefdblibrary provides out of the box.gateauxdoes not handle FoundationDB connections for you, just the data parsing
part of your application. Effectivelygateauxis just a data schema enforcing fancy
wrapper that sits on top of tuple packing and unpacking with some nice syntactic sugar.gateauxdoes not abstract away any of the useful existingfdbkeyspace interface.While there is overhead in checking data and converting it between types,gateauxis
relatively performant as all it does is shuffle native Python data types about.Installationgateauxitself only depends onfoundationdbandpytz. You first need to install
the FoundationDB client libraries from:https://www.foundationdb.org/downloadand FoundationDB Python library from PyPI:$pipinstallfoundationdbNext, installgateauxfrom PyPI:$pipinstallgateauxPython >= 3.6 is required due to the use of typing.ExamplesGeneral example usageFoundationDB class scheduling example rebuilt with gateauxFoundationDB temperature readings example rebuilt with gateauxEnforced data formatgateauxenforces certain requirements. These are not suitable for every project so
check carefully and verify the library is appropriate for your application before you
use it:All structures are in their own FoundationDB subspace, but it is up to you what
keyspace or subspace to useKey tuple members are variable, a key of 3 elements can contain 1, 2 or 3 values,
this is to support prefixes and ranges for keysValue tuple members are fixed, a value of 3 elements must always contain 3 valuesValidation is strict, if you define a field as a StringField you cannot store bytes
in it etc.While possible to support multiple data types, such as cast int(1) to str('1') if an
integer is provided to a StringField, by design typing is enforced and this will
raise an exceptionYou should not use direct binary data with FoundationDB while usinggateaux, always
use tuples of other types,tr[b'key'] = b'valueis incompatible withgateaux, buttr[(b'key',)] = (b'value',)is compatible.gateauxconforms to the same idea offdb.tuplesuch that packing then unpacking a
tuple should always result in the same original tuple of data.StructuresThere is only one base structure which is inherited to create your own structures.
Synopsis:importfdbimportgateauxfdb.api_version(620)db=fdb.open()classSomeUserStructure(gateaux.Structure):key=(gateaux.BinaryField(),)value=(gateaux.BinaryField(),)some_keyspace=fdb.Subspace(('some','subspace'))some_structure_instance=SomeUserStructure(some_keyspace)Structures have one required argument, a FoundationDB subspace. Structure instances
have the following interface for tuples:structure.pack_key((...))validates a tuple of data against the defined key
fields and returns bytes. The bytes are a FoundationDB packed tuple in the defined
directory.structure.unpack_key(b'...')unpacks FoundationDB bytes into a tuple and then
validates the data against the defined key fields returning the appropriate data type
for the field.structure.pack_value((...))validates a tuple of data against the defined value
fields and returns bytes. The bytes are a FoundationDB packed tuple in the defined
directory.structure.unpack_value(b'...')unpacks FoundationDB bytes into a tuple and then
validates the data against the defined value fields returning the appropriate data
type for the field.And the following interface for dicts:structure.pack_key_dict({...})validates a dict of data against the defined key
fields and returns bytes. The bytes are a FoundationDB packed tuple in the defined
directory. To use key dicts you must have given all of your key fields a name.structure.unpack_key_dict(b'...')unpacks FoundationDB bytes into a dict and then
validates the data against the defined key fields returning the appropriate data type
for the field. To use key dicts you must have given all of your key fields a name.structure.pack_value_dict((...))validates a dict of data against the defined value
fields and returns bytes. The bytes are a FoundationDB packed tuple in the defined
directory. To use value dicts you must have given all of your value fields a name.structure.unpack_value_dict(b'...')unpacks FoundationDB bytes into a dict and then
validates the data against the defined value fields returning the appropriate data
type for the field. To use value dicts you must have given all of your value fields a
name.And the following properties:structure.descriptiona property which returns adictdescribing the model,
including the name of the structure and any doc string as well as lists of
descriptions for each field in the key and value. You can use this to programmatically
inspect a structure in the future and is useful if you have many structures.FieldsAll fields support the following arguments:name=stringIf set it defines a name stored for the field.help_text=stringIf set, help defines some optional help text to describe the data
stored in the value.null=booleanIf set to True then the field can have aNonevalue. Defaults to
False.default=valueIf set, defines a default for a field. The type must match the
required type for the field. A default is only used ifnull=TrueandNoneis
provided to the field.Other field types may support more arguments.BinaryFieldStores bytes. Optional arguments:max_length=intIf set defines the maximum number of bytes the field will store.Accepted type:bytesIntegerFieldStores integers. Optional arguments:min_value=intIf set defines the minimum number the field will acceptmax_value=intIf set defines the maximum number the field will acceptAccepted type:intFloatFieldStores floats. Optional arguments:min_value=floatIf set defines the minimum number the field will acceptmax_value=floatIf set defines the maximum number the field will acceptAccepted type:floatBooleanFieldStores booleans.Accepted type:boolStringFieldStores strings. Optional arguments:max_length=intIf set defines the maximum length of the field will acceptAccepted type:strDateTimeFieldStores datetime instances. Internally stored as a UNIX timestamp as floats in UTC.Accepted type:datetime.datetimeInputdatetime.datetimewill be normalised to UTC and returned in UTC when read. You
should account for this in your application. Storing adatetime.datetimein any other
timezone will convert it to UTC when read. If the provideddatetime.datetimeinstance
has no timezone info it will be assumed to be UTC.IPv4AddressFieldStores IPv4 addresses. Internally stored as 4 bytes.Accepted type:ipaddress.IPv4AddressIPv4NetworkFieldStores IPv4 networks. Internally stored as 5 bytes (address + prefix length).Accepted type:ipaddress.IPv4NetworkIPv6AddressFieldStores IPv6 addresses. Internally stored as 16 bytes.Accepted type:ipaddress.IPv6AddressIPv6NetworkFieldStores IPv4 networks. Internally stored as 17 bytes (address + prefix length).Accepted type:ipaddress.IPv6NetworkEnumFieldStores Enums. Internally stored as an integer that maps to a specified value. Required
arguments:members=tupleA mandatory tuple of ints to use as Enum members.Example:MEMBER_A=0MEMBER_B=1MEMBERS=(MEMBER_A,MEMBER_B)EnumField(members=MEMBERS)Accepted type:intUUIDFieldStores UUID instances. Internally stored as 16 bytes.Accepted type:uuid.UUIDTestsThere is a pretty comprehensive test suite. Asgateauxis designed to pack and unpack
important data it has good coverage to make sure it's behaving as expected. You can run
it by cloning this repository then executing:$./run-tests.shThe tests perform type checking and require "mypy" fromhttp://mypy-lang.org/to be
globally installed. E.g.apt install mypy.ContributingAll properly formatted and sensible pull requests, issues and comments are welcome.