Package: redis-helper
Note: as of redis-helper v0.4.0, version 3.0 of redis-py is in use,
which has backwards incompatible changes withe redis-py 2.x. Seehttps://github.com/redis/redis-py/tree/70ef9ec68f9163c86d4cace2941e2f0ae4ce8525#upgrading-from-redis-py-2x-to-30AboutInstall redis-helper, create an instance ofredis_helper.Collection(the args/kwargs define the model) and use theadd,get,update,delete, andfindmethods to:quickly store/retrieve/modify Python dicts in Redisfilter through indexed fields with simple/flexible find argumentspower real-time dashboards with metrics at a variety of time rangessuper-charge event logging and system debuggingbuild FAST prototypes and simulatorsgreatly simplify data access patterns throughout applicationSee therequest logging demoandurls
demo(withunique_fielddefined). Theexamplesthey reference areshortandeasy to read.Theredis-helper projectevolved from areference Python
projectthat would beeasy to teachand follow many practical best practices
and useful patterns. Main purpose was to have something that was supereasy to configure(a single~/.config/redis-helper/settings.inifile for multiple application environments) that did cool things withRedis.Theredis-helper packageprovides aCollectionclass that was designed to beeasy to
interact within the shell (for exploration, experimentation, and
debugging). Most methods on aCollectionhelpminimize typing(passing multiple arguments in a single delimited string when
appropriate) and do “the most reasonable thing” whenever possible.The first time thatredis_helperis imported, the samplesettings.inifile will be copied to the~/.config/redis-helperdirectory.[default]
image_version = 6-alpine

[dev]
container_name = redis-helper
port = 6379
rm = False
redis_url = redis://localhost:6379/1

[test]
container_name = redis-helper-test
port = 6380
rm = True
redis_url = redis://localhost:6380/9If docker is installed to your system and your user has permission to
use it, thebg-helper docker
toolswill be used to start a redis container for development or running
tests, if Redis is not already installed locally.(Optionally) install Redis and start server locally% sudo apt-get install -y redis-server

or

% brew install redis
% brew services start redisInstall redis-helperinstall latest tag/release ofredis-helper
package% pip3 install redis-helperor, install latest commit on master ofredis-helper
project% pip3 install git+git://github.com/kenjyco/redis-helperIntroRedisis a fast in-memorydata structure server, where each stored object is referenced by a
key name. Objects in Redis correspond to one of several basic types,
each having their own set of specialized commands to perform operations.
Theredis Python packageprovides theStrictRedisclass, which contains methods that correspond to all of the Redis server
commands.When initializing Collection objects, you must specify the “namespace”
and “name” of the collection (which are used to create the internally
used_base_keyproperty). All Redis keys associated with a
Collection will have a name pattern that starts with the_base_key.importredis_helperasrhrequest_logs=rh.Collection('log','request',index_fields='status, uri, host',json_fields='request, response, headers')urls=rh.Collection('web','url',unique_field='name',index_fields='domain, _type')notes=rh.Collection('input','note',index_fields='topic, tag',insert_ts=True)sample=rh.Collection('ns','sample',unique_field='name',index_fields='status',json_fields='data',rx_name='\S{4,6}',rx_status='(active|inactive|cancelled)',rx_aws='[a-z]+\-[0-9a-f]+',insert_ts=True)uses_sample=rh.Collection('ns','uses_sample',index_fields='z',rx_thing='\S{4,6}',reference_fields='thing--ns:sample')aunique_fieldcan be specified on a collection if items in the
collection should not contain duplicate values for that particular
fieldtheunique_fieldcannot also be included injson_fieldsorpickle_fieldsif you specify aunique_field, that field must exist on each
item you add to the collectionuseindex_fieldsto specify which fields you will want to filter
on when using thefindmethodthe values for data fields being indexed MUST be simple strings or
numbersthe values for data fields being indexed SHOULD NOT be long
strings, as the values themselves are part of the index keysusejson_fieldsto specify which fields should be JSON encoded
before insertion to Redis (using the very fastujsonlibrary)userx_{field}to specify a regular expression for any field with
strict rules for validationusereference_fieldsto specify fields that reference theunique_fieldof another collectionuses field–basekey combosusepickle_fieldsto specify which fields should be pickled
before insertion to Redissetinsert_ts=Trueto create an additional index to store insert
timesonly do this if you are storing items that you are likely to
update and also likely to want to know the original insert timeeach time an object is updated, the score associated with thehash_id(at the_ts_zset_key) is updated to the current
timestampthe score associated with thehash_id(at the_in_zset_key) is never updatedEssentially, you can store a Pythondictin a Redishashand
index some of the fields in Redissets. The collection’s_ts_zset_keyis the Redis key name for thesorted
setcontaining thehash_idof every hash in the collection (with thescorebeing autc_floatcorresponding to the UTC time thehash_idwas added or
modified).ifinsert_ts=Truewas passed in when initializing theCollection(or sub-class), then the collection will also defineself.in_zset_keyto be the Redis key name for the sorted set (forhash_idandutc_floatof insert time)request_logs.add(method='get',status=400,host='blah.net',uri='/info',request={'x':50,'y':100},response={'error':'bad request'},)urls.add(name='redis-helper github',url='https://github.com/kenjyco/redis-helper',domain='github.com',_type='repo',)Thegetmethod is a wrapper tohash
commandshget,hmget, orhgetall. The actual hash command that gets called is determined by
the number of fields requested.a Python dict is typically returned fromgetifitem_formatis specified, a string will be returned matching
that format insteadrequest_logs.get('log:request:1')request_logs.get('log:request:1','host,status')request_logs.get('log:request:1',item_format='{status}for{host}{uri}')request_logs.get_by_position(0,item_format='{status}for{host}{uri}')urls.get_by_position(-1,'domain,url')urls.get_by_unique_value('redis-helper github',item_format='{url}points to a{_type}')theget_by_positionandget_by_unique_valuemethods are
wrappers togettheget_by_unique_valuemethod is only useful if aunique_fieldwas set on the CollectionThefindmethod allows you to return data for items in the
collection that match some set of search criteria. Multiple search terms
(i.e.index_field:valuepairs) maybe be passed in thetermsparameter, as long as they are separated by one of,;|.
Any fields specified in theget_fieldsparameter are passed along to
thegetmethod (when the actual fetching takes place).when usingterms, all terms that include the same field will be
treatead like an “or” (union of related sets), then the intersection
of different sets will be computedsee the Redisset commandsandsorted set commandsThere are many options for specifying time ranges in thefindmethod
including:sinceanduntilwhen specifyingnum:unitstrings
(i.e. 15:seconds, 1.5:weeks, etc)start_tsandend_tswhen specifying timestamps with a form
betweenYYYYandYYYY-MM-DDHH:MM:SS.fstartandendwhen specifying autc_floatforsince,until,start_ts, andend_ts, multiple
values may be passed in the string, as long as they are separated by
one of,;|.when multiple time ranges are specified, thefindmethod will
determine all reasonable combinations and return a result-set per
combination (instead of returning a list of items, returns a dict
of list of items)Ifcount=Trueis specified, the number of results matching the
search criteria are returned instead of the actual resultsif there are multiple time ranges specified, counts will be returned
for each combinationrequest_logs.find('status:400, host:blah.net',get_fields='uri,error')request_logs.find(since='1:hr, 30:min',until='15:min, 5:min')request_logs.find(count=True,since='1:hr, 30:min',until='15:min, 5:min')urls.find(count=True,since='1:hr, 30:min, 10:min, 5:min, 1:min')urls.find(start_ts='2017-02-03',end_ts='2017-02-03 7:15:00')urls.find(start_ts='2017-02-03',item_format='{_ts}->{_id}')Theupdatemethod allows you to change values for some fields
(modifying theunique_field, when it is specified, is not allowed).every time a field is modified for a particularhash_id, the
previous value and score (timestamp) are stored in a Redis hashtheold_data_for_hash_idorold_data_for_unique_valuemethods
can be used to retrieve the history of all changes for ahash_idurls.update('web:url:1',_type='fancy',notes='this is a fancy url')urls.old_data_for_hash_id('web:url:1')urls.old_data_for_unique_value('redis-helper github')Theload_ref_dataoption onget,get_by_unique_value, orfindmethods allow you to load the referenced data object from the
other collection (wherereference_fieldsare specified)In[1]:sample.add(name='hello',aws='ami-0ad5743816d822b81',status='active')Out[1]:'ns:sample:1'In[2]:uses_sample.add(thing='hello',z=500,y=True)Out[2]:'ns:uses_sample:1'In[3]:uses_sample.get('ns:uses_sample:1')Out[3]:{'thing':'hello','z':500,'y':True}In[4]:uses_sample.get('ns:uses_sample:1',load_ref_data=True)Out[4]:{'thing':{'name':'hello','aws':'ami-0ad5743816d822b81','status':'active','_id':'ns:sample:1','_ts':20201028210044.875},'z':500,'y':True}In[5]:uses_sample.add(thing='byebye',z=100,y=True)Out[5]:'ns:uses_sample:2'In[6]:uses_sample.get('ns:uses_sample:2',load_ref_data=True)Out[6]:{'thing':'byebye','z':100,'y':True}TipThere may be times where you want to use redis-helper (if it’s already
installed), but don’t want to make it an explicit requirement of your
project. In cases like this you can do the following:try:
    import redis_helper as rh
    from redis import ConnectionError as RedisConnectionError
except (ImportError, ModuleNotFoundError):
    SomeCollection = None
else:
    try:
        SomeCollection = rh.Collection(
            ...
        )
    except RedisConnectionError:
        SomeCollection = NoneThen in whatever function, you can just do:def some_func():
    if SomeCollection is None:
        return

    # Do stuff with SomeCollectionLocal development setup% git clone https://github.com/kenjyco/redis-helper
% cd redis-helper
% ./dev-setup.bashThedev-setup.bashscript will create a virtual environment in the./venvdirectory
with extra dependencies (ipython, pdbpp, pytest), then copysettings.inito the~/.config/redis-helperdirectory.Running tests in development setupThesetup.cfgfile contains the options forpy.test, currently-vsx-rs--pdb.The-vsx-rs--pdboptions will run tests in a verbose manner and
output the reason why tests were skipped (if any were skipped). If there
are any failing tests,py.testwill stop on the first failure and
drop you into apdb++debugger
session.See thedebugging
sectionof the README for tips on using the debugger and setting breakpoints (in
the actualproject
code,
or in thetest
code).% venv/bin/py.testor% venv/bin/python3 setup.py testNote: This option requiressetuptoolsto be installed.UsageTherh-download-examples,rh-download-scripts,rh-notes, andrh-shellscripts are provided.$ venv/bin/rh-download-examples --help
Usage: rh-download-examples [OPTIONS] [DIRECTORY]

  Download redis-helper example files from github

Options:
  --help  Show this message and exit.

$ venv/bin/rh-download-scripts --help
Usage: rh-download-scripts [OPTIONS] [DIRECTORY]

  Download redis-helper script files from github

Options:
  --help  Show this message and exit.

$ venv/bin/rh-notes --help
Usage: rh-notes [OPTIONS] [TOPIC]

  Prompt user to enter notes (about a topic) until finished; or review notes

Options:
  -c, --ch TEXT  string appended to the topic (default "> ")
  -s, --shell    Start an ipython shell to inspect the notes collection
  --help         Show this message and exit.

$ venv/bin/rh-shell --help
Usage: rh-shell [OPTIONS]

  Interactively select a Collection model and start ipython shell

Options:
  --help  Show this message and exit.>>>importredis_helperasrh>>>collection=rh.Collection(...,index_fields='field1, field3')>>>hash_id=collection.add(field1='',field2='',field3='',...)>>>collection.add(...)>>>collection.add(...)>>>collection.update(hash_id,field1='',field4='',...)>>>change_history=collection.old_data_for_hash_id(hash_id)>>>data=collection.get(hash_id)>>>some_data=collection.get(hash_id,'field1, field3')>>>results=collection.find(...)>>>results2=collection.find('field1:val, field3:val',...)>>>results3=collection.find(...,get_fields='field2, field4')>>>counts=collection.find(count=True,...)>>>top_indexed=collection.index_field_info()>>>collection.delete(hash_id,...)Basics - Part 1 (request logging demo)Demobookmarks:1:10is when theipythonsession is started withvenv/bin/ipython-irequest_logs.py3:14is when a secondipythonsession is started (in a separate tmux pane) to simulate
a steady stream of requests withslow_trickle_requests(randomsleep=True, show=True)4:22is when theindex_field_infomethod is used to get the latest counts of top
indexed items6:11is whenslow_trickle_requests(.001)is run to simulate a large quick
burst in traffic7:00is when multiple
values are passed in thesinceargument offind…request_logs.find(count=True,since='5:min,1:min, 30:sec')8:37is whengetandget_by_positionmethods are used with a variety of arguments to
change the structure of what’s returned10:33is when theredis_helper.ADMIN_TIMEZONEis changed at run time fromAmerica/ChicagotoEurope/London11:27is whenfindis used with a variety of arguments to change the structure of what’s
returned14:30is whenfindis used with multiple search terms and multiplesincevalues…request_logs.find('host:dogs.com,uri:/breeds',count=True,since='5:min,1:min, 10:sec')15:54is when theupdatemethod is used to modify data and change history is
retrieved with theold_data_for_hash_idmethodThe first demo walks through the following:creating a virtual environment, installing redis-helper, and
downloading example files$ python3 -m venv venv
$ venv/bin/pip3 install redis-helper ipython
$ venv/bin/rh-download-examples
$ cat ~/.config/redis-helper/settings.ini
$ venv/bin/ipython -i request_logs.pyusing the sampleCollectiondefined inrequest_logs.pytoshow values of some properties on aCollectionredis_helper.Collection._base_keyredis_helper.Collection.now_prettyredis_helper.Collection.now_utc_floatredis_helper.Collection.keyspaceredis_helper.Collection.sizeredis_helper.Collection.firstredis_helper.Collection.lastshow values of some settings fromredis_helperredis_helper.APP_ENVredis_helper.REDIS_URLredis_helper.REDISredis_helper.SETTINGS_FILEredis_helper.ADMIN_TIMEZONEshow output from some methods on aCollectionredis_helper.Collection.index_field_info()redis_helper.Collection.find()redis_helper.Collection.find(count=True)redis_helper.Collection.find(count=True,since='30:sec')redis_helper.Collection.find(since='30:sec')redis_helper.Collection.find(since='30:sec',admin_fmt=True)redis_helper.Collection.find(count=True,since='5:min,1:min, 30:sec')redis_helper.Collection.find('index_field:value')redis_helper.Collection.find('index_field:value',all_fields=True, limit=2)redis_helper.Collection.find('index_field:value',all_fields=True, limit=2, admin_fmt=True,item_format='{_ts}->{_id}')redis_helper.Collection.find('index_field:value',get_fields='field1,field2', include_meta=False)redis_helper.Collection.find('index_field1:value1,index_field2:value2', count=True)redis_helper.Collection.find('index_field1:value1,index_field2:value2', count=True,since='5:min,1:min, 10:sec')redis_helper.Collection.get(hash_id)redis_helper.Collection.get(hash_id, 'field1,field2,field3')redis_helper.Collection.get(hash_id, include_meta=True)redis_helper.Collection.get(hash_id, include_meta=True,fields='field1,field2')redis_helper.Collection.get(hash_id, include_meta=True,item_format='{_ts}->{_id}')redis_helper.Collection.get_by_position(0)redis_helper.Collection.get_by_position(0, include_meta=True, admin_fmt=True)redis_helper.Collection.update(hash_id,field1='value1',field2='value2')redis_helper.Collection.old_data_for_hash_id(hash_id)Basics - Part 2 (urls demo, with unique field)Demobookmarks:TODOThe second demo walks through the following:using the sampleCollectiondefined inurls.pytoTODOSettings, environments, testing, and debuggingTo trigger a debugger session at a specific place in theproject
code,
insert the following, one line above where you want to inspectimport pdb; pdb.set_trace()To start the debugger insidetest
code, usepytest.set_trace()use(l)istto list context linesuse(n)extto move on to the next statementuse(s)tepto step into a functionuse(c)ontinueto continue to next break point
(i.e.set_trace()lines in your code)usestickyto toggle sticky mode (to constantly show the
currently executing code as you move through with the debugger)useppto pretty print a variable or statementIf the redis server atredis_url(in thetest sectionof~/.config/redis-server/settings.ini) is not running or is not empty,
redis server tests will be skipped.Use theAPP_ENVenvironment variable to specify which section of thesettings.inifile your settings will be loaded from. Any settings in
thedefaultsection can be overwritten if explicity set in another
section.if noAPP_ENVis explicitly set,devis assumedtheAPP_ENVsetting is overwritten to betestno matter what
was set when callingpy.testtests