Package: psense-common
PSENSE-COMMONCommon Modules used by internal PercuSense python applications. This project follows semantic versioning (breaking.major.minor release).Getting StartedGeneral InstructionsSteps to try this out yourself:Install thepsense-commonlibrary:From PyPi$pipinstallpsense-commonLocal Installation$gitclonegit@bitbucket.org:psense/psense-common.git
$pythonsetup.pyinstallImport the modules you need in your script.frompsense_commonimportPSenseParser)Access module methods directly.frompsense_commonimportPSenseParserfilename='/path/to/data/file'parser=PSenseParser()parser.identify_file_source(filename)ifparser.source:parser.load_rawfile(filename)print(parser.data)# this is of type pandas.DataFrameAWSpsense_common/psense_aws_itfc.pyintroduces the classPSenseAWSInterface, which provides an interface to the Amazon DynamoDB datastore. Currently, data is stored on a per-record basis (NoSQL) for Experiment, Sensor, and Event data ("Experiments", "SensorData", and "Calibration" -- not fully supported, will be changed to "EventData" in migration from 0.x.x to 1.x.x).The class contains query and put operations but does not provide provisions for deleting items from the datastore.CredentialsAt initialization, the class will create an AWS session that is persisted. The session object allows the user to authenticate just a single time (rather than for every communication).In generating the session, bot3 will attempt to load a profile (if not provided, will try "PShield") that contains the necessary credentials for authentication/authorization to our dynamodb instances. If the profile doesn't exist or cannot be loaded, the class reverts to the environment default AWS credentials in the 'us-east-1' region.configandcredentialfiles for aws are stored in the user's home directory:~/.aws/.  In Windows, the comparable location isC:\Users\[username]\.aws\UsageGet Sensor Datafrompsense_commonimportPSenseAWSInterfaceimportpytzaws=PSenseAWSInterface(debugmode=True)aws.set_query_config(req_size=7200,query_count=2)experiments=['[my experiment id 1]','[my experiment id 2]']forexpidinexperiments:ifnotaws.verif_experiment(expid):print('invalid experiment id skipping{}'.format(expid))passcount,data=aws.get_sensor_data()data.index=data.index.tz_localize(pytz.utc).tz_convert(localtz)print(data)Add Experiment and real-time Sensor datafrompsense_commonimport(PSenseAWSInterface,PSenseParser)# user variablesfilename='[vfp600 gamry file].txt'expid='experimentid'# initialize classes + helper funcparser=PSenseParser(debugmode=True)aws=PSenseAWSInterface(debugmode=True)aws.set_query_config(req_size=7200,query_count=2)deftail(fin):"Listen for new lines added to file."whileTrue:where=fin.tell()line=fin.readline()ifnotline:time.sleep(SLEEP_INTERVAL)fin.seek(where)else:yieldline# identify the type of file we are parsingifnotparser.identify_file_source(filename):print('unknown file type. exiting')assertFalse# confirm that the experiment id is validifnotaws.verif_experiment(expid):print('invalid experiment id skipping{}'.format(expid))assertFalse# add experiment to databaseadd_experiment_success,if_fail_reason=aws.add_experiment()ifif_fail_reason=='error':print('Aborted. Error occurred in communication with AWS.')assertFalseelifif_fail_reason=='exists':print('Experiment already exists in database. Must delete existing experiment sensor data before uploading new values.')# "tail" the file and send each record to ddbwithopen(filename,'r')asfin:# skip to the end of the file before beginning tailfin.seek(0,os.SEEK_END)forlineintail(fin):row=parser.parse_record(line.strip())aws.add_sensordata(*row)Future notes: Sensor and Event data should remain as a blob-store -- we expect schema to change depending on the type of sensor (or event). For example, sensors with 3 working electrodes should contain more properties than sensors with a single signal.Experiment ID FormattingAll sensor data is associated with a particular experiment ID. Experiment IDs should be unique and follow the PercuSense naming scheme.UsageDecoding an experiment:frompsense_commonimport(psense_format)experiment='[my experiment id]'validate_name=psense_format.psense_exp_title()validate_name.decode(experiment)Generating an experiment id through console IO (must provide 3-digit device id):frompsense_commonimport(psense_format)device_id='P01'experiment=psense_format.setup_new_experiment(device_id)Data ParsingThe PSenseParser class will parse data from flat-file into Pandas DataFrame. Output object will contain columns appropriate for PercuSense analysis scripts (and PercuSense Data Viewer web application). The parser works on individual records (live streaming) as well as full files (retrospective analysis).Supported formats:BWII2-channel
BWII3-Channel("BWII-MINI")PSHIELD
GAMRYVFP600
GAMRYEXPLAIN
DATAVIEWERsingle-channel
DATAVIEWER2-channel
DATAVIEWER3-channel
CHInstruments(txt)TODO: documentation pendingCheckdemo/parser.pyfor some example usage.See above sections (General Instructions, AWS) for basic usage.PercuSense Data FilterTODO: documentation pendingDevelopmentTests$pythonsetup.pytest.. or with code coverage (pip install --upgrade coverage):$coveragerun--source=psense_common/setup.pytest$coveragereport-mPublishingBitbucket has been configured to run tests and publish directly to pypi after code is merged to master. Seebitbucket-pipelines.ymlfor configuration details.Manual publishing (not recommended):$ rm -rf dist
$ python setup.py sdist
$ twine upload dist/*