Package: python-disk-collections
Module contains class with extended python list that stores items at disk.
By default items before save are pickled and compressed. Use that list
as usual list!In addition, there is implemented extended python deque with disk storage and
same behaviour ascollections.deque.Intend of package was to create generic iterables that stores really big collection of items
that does not fit in memory and to avoid usage of external cache and local database
storages.>>>fromdiskcollections.iterablesimportFileList>>>flist=FileList()>>>flist.extend([1,2,3])>>>flist.append(4)>>>flist[1,2,3,4]>>>flist[2]3>>>flist2=flist[:]# copy makes new FileList>>>my_list=list(flist)# now its simple list>>>fromdiskcollections.iterablesimportFileDeque>>>fdeque=FileDeque()>>>fdeque.extend([1,2,3])>>>fdeque.append(4)>>>fdequeFileDeque([1,2,3,4])>>>fdeque.pop()4>>>fdeque.appendleft(0)>>>fdeque.popleft()0There are available more ways to serialize items.>>>fromdiskcollections.iterablesimportList,FileList,FileDeque>>>fromdiskcollections.serializersimport(PickleSerializer,# pickle itemsPickleZLibSerializer,# pickle + compress itemsJsonSerializer,# convert to json itemsJsonZLibSerializer# convert to json + compress items)>>>fromfunctoolsimportpartial>>>JsonFileList=partial(List,serializer_class=JsonHandler)>>>flist=JsonFileList()>>>flist.append({'a':1,'b':2,'c':3})>>>flist[0]{u'a':1,u'b':2,u'c':3}InstallationTo install package type$pipinstallpython-disk-collectionsHow it worksExplaining example above:>>>fromdiskcollections.iterablesimportFileList>>>fromdiskcollections.serializersimportJsonZLibSerializer>>>>>>flist=FileList(serializer_class=JsonZLibSerializer)New instance of this object creates new temporary directory.
By usingserializer_class=JsonZLibSerializereach incoming item to list will be: json.dumped and compressed>>>flist.append({'a':1,'b':2,'c':3})so using this serializer have in mind that all objects you put into list
have to lend themself and compatible with json.
Exactly this object{‘a’: 1, ‘b’: 2, ‘c’: 3}will serialized and compressed and saved inside temporary directory.>>>flist[0]{u'a':1,u'b':2,u'c':3}Getting an item will read a file and becauseJsonZLibSerializeris used: then content will be decompressed and tried
to loaded from json.This package provides a few other serializers:PickleSerializer - pickle itemsPickleZLibSerializer - pickle + compress itemsJsonSerializer - convert to json itemsJsonZLibSerializer - convert to json + compress itemsfromdiskcollections.serializersimport(PickleSerializer,PickleZLibSerializer,JsonSerializer,JsonZLibSerializer,)In order to implement your serializer create class with methods:dumpsandloadsor import interface.>>>fromdiskcollections.interfacesimportISerializerclassISerializer:@staticmethoddefdumps(obj):"""Converts object to string.

    :param obj: any python object
    :return: dumped string
    """raiseNotImplementedError@staticmethoddefloads(obj):"""Restored dumped string into python object.

    :param obj: Object stored as string
    :return: python object restored from dump
    """raiseNotImplementedErrorAll serializers from example above implements interfaceISerializer.Under the hood,FileListfor storage items usestempfile.mktemp(in python2)
ortempfile.TemporaryDirectory(in python3). It means, that every list
has own unique directory, placed likely in/tmp/.
When list is removed by garbage collector, all items that was stored are lost.ForFileDequestores items in the same way asFileList.By default on exit program, or when list or deque is removed: all content of files also are dropped.To prevent this usePersistentDirectoryClient:>>>fromfunctoolsimportpartial>>>fromdiskcollections.iterablesimportList,PersistentDirectoryClient>>>fromdiskcollections.serializersimportJsonSerializer>>>fromdiskcollections.iterablesimportPersistentDirectoryClient>>>dir_abc=partial(PersistentDirectoryClient,"abc")>>>persistent_list=List(client_class=dir_abc,serializer_class=JsonSerializer)>>>persistent_list.append({"a":1,"b":2})>>>assertlen(persistent_list)==1>>>assertopen("abc/0").read()=='{"a": 1, "b": 2}'On exit directoryabcwith file0of his contents will still exist.ContributeFork repository on GitHub to start making your changes to themasterbranch (or branch off of it).Write tests that prove that bug or future works as expectedInstall other python versions withpyenvtogether withtox:$sudoapt-getinstallpyenvtoxInstall other python versions$pyenvinstall2.73.53.63.73.83.93.103.11Make them global fordetoxpackage$pyenvglobal2.73.53.63.73.83.93.103.11Install globallydetox$sudopipinstalldetoxCheck your code and tests withdetox$detox-n1GLOBsdist-make:python-disk-collections/setup.pylintinst-nodeps:python-disk-collections/.tox/.tmp/package/7/python-disk-collections-0.0.4.ziplintrun-test-pre:PYTHONHASHSEED='1334400931'lintruntests:commands[0]|flake8lintruntests:commands[1]|pythonsetup.pycheck-r-s-mpy27inst-nodeps:python-disk-collections/.tox/.tmp/package/7/python-disk-collections-0.0.4.zippy27run-test-pre:PYTHONHASHSEED='1334400931'py27runtests:commands[0]|py.test-v--covdiskcollections--cov-config.coveragerc--cov-reportterm-missing--cov-fail-under95...py311inst-nodeps:python-disk-collections/.tox/.tmp/package/7/python-disk-collections-0.0.4.zippy311run-test-pre:PYTHONHASHSEED='1334400931'py311runtests:commands[0]|py.test-v--covdiskcollections--cov-config.coveragerc--cov-reportterm-missing--cov-fail-under95_________________________________________________________________________________________________________________summary__________________________________________________________________________________________________________________lint:commandssucceededpy27:commandssucceededpy35:commandssucceededpy36:commandssucceededpy37:commandssucceededpy38:commandssucceededpy39:commandssucceededpy310:commandssucceededpy311:commandssucceededcongratulations:)Send a pull request!LicensePython-Disk-Collection is under MIT license, see LICENSE for more details.