Package: pcas
PCAS Python interfaceThepcasPython package provides an interface to services running onPCASinfrastructure. It consists of several submodules:keyvalueloggersqlKeyvalue interfaceThekeyvaluesubmodule provides an interface to key-value databases hosted on servers running the PCAS databasekvdbd. To connect to such a database, use aConnectionobject.Connection objectsAConnectionrepresents a connection to a PCAS key-value database. You can create aConnectionlike this:frompcasimportkeyvaluedb='...your database name...'ssl_cert='...your SSL certificate...'conn=keyvalue.Connection(db,address='localhost:12356',certificate=ssl_cert)Theaddressparameter is optional; if omitted, its value will be read from the environment variablePCAS_KVDB_ADDRESS. Similarly if thecertificateparameter is omitted, its value will be read from the environment variablePCAS_SSL_CERT.AConnectionobject has methods to connect to a table, create a new table, delete a table, describe a table, and list the tables in the database.Connect to a tableTo connect to a table, use theconnect_to_tablemethod. This takes the name of the table as an argument, and returns aTableobject.t=conn.connect_to_table('table_name')Tableobjects are documentedbelow.Create a tableTo create a new table, use thecreate_tablemethod. This takes the name of the table and a dictionary as an argument, and returnsNone. The dictionary represents a template key-value record: it should have keys that are strings and values that are strings, integers, booleans, floating-point numbers, orbytesobjects.conn.create_table('new_table',template)Delete a tableTo delete a table from the database, use thedelete_tablemethod. This takes the name of the table as an argument, and returnsNone.conn.delete_table('unwanted')Describe a tableTo get a description of a table in the database, use thedescribe_tablemethod. This takes the name of the table as an argument, and returns a dictionary describing a typical entry in the table.d=conn.describe_table('table_name`)The return value is a dictionary such as{'id':0,'coefficients':''}describing a typical key-value record in the table; here we see that the value of'id'in the records is an integer, and the value of'coefficients'is a string. If the database backingkvdbdhas a schema (e.g. PostgreSQL) then the table description uses this schema; if the backing database is schemaless (e.g. MongoDB) then the table description is a "best guess" based on a sample of records from the table, and may not be correct for all records.List tablesTo list the tables in the database, use thelist_tablesmethod. The takes no arguments, and returns a list of the names of the tables in the database.table_list=conn.list_tables()Rename a tableTo rename a table in the database, use therename_tablemethod. This takes the old and new names as arguments, and returnsNone.# List the tablesprint(conn.list_tables())# {'hippo'}# Rename the tableconn.rename_table('hippo','giraffe')# List the tables againprint(conn.list_tables())# {'giraffe'}Table objectsATableobject represents a table in a PCAS keyvalue database. As discussed, you create aTableby calling theconnect_to_tablemethod on theConnectionthat represents the database.Tableobjects have methods to count, delete, insert, select, and update key-value records in the table, as well as methods to add, list, and delete indexes on keys of the key-value records.Count recordsTo count records in the table, use thecountmethod. It optionally takes a selector as argument, and returns the number of records in the table that match that selector.n=t.count(selector)The selector can be omitted, in which case the total number of records in the table is returned; it can be a dictionary, representing a key-value record, in which case the number of records matching those keys and values is returned; or it can be a string inSQL formatthat specifies the condition. Examples:# Count all the records in the table tnum_records=t.count()# Count the records with the specified keys and valuesselector={'rank':3,'name':'exactly this'}n=t.count(selector)# Count the records matching the specified conditionn=t.count('WHERE rank BETWEEN 2 and 4 AND id >= 100')Delete recordsTo delete records from the table, use thedeletemethod. It optionally takes a selector as argument, and returns the number of records deleted.num_deleted=t.delete(selector)The selector can be omitted, in which case all records in the table are deleted; it can be a dictionary, representing a key-value record, in which case all records matching those keys and values are deleted; or it can be a string inSQL formatthat specifies the condition. Examples:# Delete all the records in the table tnum_deleted=t.delete()# Delete all records with the specified keys and valuesselector={'id':3,'rank':5}num_deleted=t.delete(selector)# Delete all records that match the specified conditionnum_deleted=t.delete('WHERE id = 3 AND rank <= 6')Insert recordsTo insert records into the table, use theinsertmethod. It takes an iterable of dictionaries as argument, and returnsNone. The dictionaries represent key-value records; they should have keys that are strings and values that are strings, integers, booleans, floating-point numbers, orbytesobjects.t.insert(recs)Select recordsTo select records from the table, use theselectmethod. It takes a dictionary, an optional selector, an optional sort order, and an optional limit as arguments, and returns an iterator containing the matching records.itr=t.select(template,selector,order,limit)The records returned will be in the form specified by the key-value recordtemplate. If anorderis provided, the records will be returned in that order. If alimitis provided, it must be a positive integer and at most that many records will be returned. The selector can be omitted, in which case all records in the table are returned; it can be a dictionary, representing a key-value record, in which case all records matching those keys and values are returned; or it can be a string inSQL formatthat specifies the condition.If the selector is provided as a string inSQL formatthen it may also specify the order and/or the limit. If it does so then order, and/or respectively limit, must either be unspecified or specified asNone. Examples:# We return the id and the rank, which are both integerstemplate={'id':0,'rank':0}# Select all the records in the table titr=t.select(template)# Select at most 10 recordsitr=t.select(template,limit=10)# Select records with specified keys and valuesselector={'rank':2,'dimension':4}itr=t.select(template,selector)# Select records that match a specified conditionitr=t.select(template,'WHERE id > 40 AND dimension IN (1,3,5,7)')# Select with the records returned in a specified orderitr=t.select(template,'WHERE id > 40 SORT BY id ASC, rank DESC')# An equivalent select, with the sort order specified directly.frompcas.sqlimportsortmy_order=[sort.ascending('id'),sort.descending('rank')]itr=t.select(template,'WHERE id > 40',order=my_order)Update recordsTo update records in the table, use theupdatemethod. It takes a dictionary and an optional selector as arguments, and returns the number of records updated.num_updated=t.update(replacement,selector)All records in the table that match the selector are updated by setting the keys present in the dictionaryreplacementto the corresponding values. The selector can be omitted, in which case all records in the table are updated; it can be a dictionary, representing a key-value record, in which case all records matching those keys and values are updated; or it can be a string inSQL formatthat specifies the condition. Examples:# We will set the values 'id' and 'dimension' in the updated recordsreplacement={'id':98,'dimension':3}# Update all the records in the table tnum_updated=t.update()# Update all records with the specified keys and valuesselector={'id':98,'rank':12}num_updated=t.update(selector)# Update all records that match the specified conditionnum_updated=t.update('WHERE id = 3 AND rank <= 6')Add, delete, and list indexesTo add an index to the table, use theadd_indexmethod. It takes the key to be indexed as an argument, and returnsNone.t.add_index('the_key_to_be_indexed')A unique index is an index together with the constraint that, for each value of the indexed key, there is at most one record with that value. To add a unique index to the table, use theadd_unique_indexmethod. It takes the key to be indexed as an argument, and returnsNone.t.add_unique_index('the_key_to_be_indexed')To delete an index from the table, use thedelete_indexmethod. It takes the key for the index as an argument, and returnsNone. If there is no index on the specified key, this operation succeeds but does nothing.t.delete_index('the_key_to_be_indexed')To list the indexes present on the table, use thelist_indicesmethod. It takes no arguments, and returns a list of the indexed keys.key_list=t.list_indices()Describe the tableTo get a description of the table, use thedescribemethod. It takes no arguments, and has return value exactly as for thedescribe_tablemethod of aConnectiondocumentedabove.d=t.describe()Add keysTo add keys to an existing table, use theadd_keysmethod. This updates each record in the table, adding the keys in the given record along with the corresponding values, if they are not already present. Any keys that are already present will be left unmodified.# Display the records in the table tprint([xforxint.select(t.describe())])# [{'a': 1, 'b': 'fish'}, {'a': 2, 'c': 'chips'}]# Add new keyst.add_keys({'b':'peas','d':12})# Display the records againprint([xforxint.select(t.describe())])# [{'a': 1, 'b': 'fish', 'd': 12}, {'a': 2, 'c': 'chips', 'b': 'peas', 'd': 12}]Delete keysTo delete keys from a table, use thedelete_keysmethod. This updates all records in the table, deleting the specified keys if present.# Display the records in the table tprint([xforxint.select(t.describe())])# [{'a': 1, 'b': 'fish', 'd': 12}, {'b': 'peas', 'd': 12, 'a': 2, 'c': 'chips'}]# Delete some keys from all recordst.delete_keys(['c','d'])# Display the records againprint([xforxint.select(t.describe())])# [{'a': 1, 'b': 'fish'}, {'a': 2, 'b': 'peas'}]Key-value recordsKey-value records are represented by dictionaries. These have keys that are strings and values that are strings, integers, booleans, floating-point numbers, orbytesobjects.Logger interfaceTheloggersubmodule provides a log handler that writes to a server running the PCAS logging servicelogd. It uses the standard Pythonlogging framework. Example usage:importmultiprocessingimportloggingfromlogging.handlersimportQueueListener,QueueHandlerfrompcasimportlogger# Set up the log handlerssl_cert='...your SSL certificate...'handler=logger.LogHandler(address='localhost:12354',certificate=ssl_cert)# Set up a queue that passes log messages to our handlerqueue=multiprocessing.Queue(-1)queue_listener=QueueListener(queue,logger.LogHandler())queue_listener.start()# Create the loggerlg=logging.getLogger('your_log_name')lg.addHandler(QueueHandler(queue))# Do stuff...# Write to the loggerlg.warning('hello world')# Clean uphandler.close()queue_listener.stop()Theaddressargument forlogger.LogHandlerspecifies the address of the PCASlogdserver and is optional; if omitted the address will be read from the environment variablePCAS_LOG_ADDRESS. Thecertificateoption specifies the SSL certificate to use and is also optional; if omitted the certificate will be read from the environment variablePCAS_SSL_CERT. The log handler blocks on logging if thelogdserver is unavailable. This is why we used a queue in the code above: it ensures that calls to the loggerlgare non-blocking.SQL submoduleThesqlsubmodule defines a parser for SQL-formatted queries. The SQL should be formatted as follows:[[WHERE]<wherecondition>][ORDERBY<sortorder>][LIMIT<limit>]See below forexamples. Note that prefixing the WHERE condition with "WHERE" is currently optional, although this might change in the future.WHEREconditionsThe following types are supported:string -	surrounded by matching double- (") or single-quotes (')integer -	must fit in a 64-bit signed or unsigned integerfloat -		must fit in a 64-bitfloatboolean -TRUEorFALSEThe following standard SQL operators are supported:=,!=<,>,<=,>=IS,IS NOTIN,NOT INBETWEEN,NOT BETWEENANDORSORTordersASORTorder should be formattedkey1[ASC|DESC],key2[ASC|DESC],...,keyn[ASC|DESC]whereASCandDESCdenote increasing and decreasing order, respectively. Precisely what this means is determined by the underlying storage engine and data type. IfASCorDESCis omitted, thenASCis assumed by default.LIMITspecificationsA non-negative integer (that must fit in 64-bit signed integer) must be provided.ExamplesWHEREis_reflexiveISFALSEANDnumber_of_points>=12ORDERBYidDESCLIMIT5WHEREidBETWEEN4and9WHERETRUEORDERBYrank,idORDERBYrankDESC,idASCLIMIT50WHERErank<=6LIMIT10