Package: arctic
Development moved to ArcticDB GitHub RepositoryThis repository and project are now in maintenance mode. Development has migrated toArcticDB.Information on how to set up, install and use Arctic has been moved toREADME-arctic.md.Changelog1.82.2 (2024-02-01)Release: #1022 Prepare release 1.82.2Perf: #1021 Improve performance of Tickstore readBugfix: #1019 Remove failing requirements.txt versionsDocs: #1019 Updated links to ArcticDB website and docsDocs: #1017 Fix doc for chunkstore's list_symbols1.82.1 (2023-10-09)Release: #1014 Prepare release 1.82.1Bugfix: #1013 Fix serialized tz name for index timezones of dateutil.timezone.tzutc()Bugfix: #1013 Fix serialized tz name for single-level index timezones of dateutil.timezone.gettz("UTC")1.82.0 (2023-07-24)Release: #1012 version 1.82.0Compatibility: update pinning from pandas<1.1, to pandas<2Compatibility: update pinning from numpy<1.19 to numpy<2Build: specify which version of python/numpy/pandas to testBugfix: #887 #928 resolve import error on pandas.Panel, Panel was removed in pandas 1.0Bugfix: fix pandas get_timezone() argument cannot be NoneBugfix: fix np.float/np.int/np.book depreciationsBugfix: fix numpy tostring() is deprecated. Use tobytes() instead.Bugfix: numpy.testing.utils is deprecatedBugfix: pandas.util.testing is deprecatedCleanup: remove async_benchmark1.81.2 (2023-05-11)Feature: #1000, #1001, #1002, #1003, #1007, #1008 ArcticDB docs, links etcBugfix:  #1009 force CircleCI to install tzlocal==4.21.81.1 (2023-03-13)Feature: #996 release wheel to pypi as well as tar.gz, egg.1.81.0 (2023-03-10)Bugfix:  #979 fix circleci mongo installFeature: #980 get testpack running with mongod 4.4.18Bugfix:  #985 suppress FutureWarning for Panel importBugfix:  #989 VersionStore slow snapshot delete add new indexBugfix:  #806, #990 DateTimeIndex tz serialization1.80.5 (2022-09-05)Feature: #950 remove all traces of python 2.7 and six packageFeature: #959 add python 3.8 supportFeature: #962 pin pandas<1.1.0, numpy<1.19.01.80.4 (2022-01-25)Bugfix:  #940 fix rows per chunk causing divide by zeroFeature: #943 check markdown rendering of README.md and CHANGES.md in buildBugfix:  #946 move test dependencies to tests_require1.80.3 (2022-01-20)Feature: #941 use named index for VersionStore to avoid 127 max fully qualified index nameBugfix:  #935, #936 fixed README.md rendering which was preventing pypi upload1.80.2 (2022-01-10)Bugfix: #932 revert serialization-optimization (#909, #910)1.80.1 (2021-12-09)Bugfix: #855 use IXSCAN for list_symbols which speeds up snapshotting (actually #856)Bugfix: #926 avoid pathologically slow count_documents() call with pymongo > 3.6.01.80.0 (2021-10-28)Feature: #919 Add CircleCI badge to README.mdFeature: #917 Add CircleCI buildBugfix:  #910 Fix column subsetting bugBugfix:  #909 Speedup FrameToArray serialiser for ChunkStore by removing intermediate DataFrame constructionBugfix:  #902 Chunkstore-read-speedupBugfix:  #872 Do not spam if not permissioned on cache dbBugfix:  #874 Pickle protocol 5 not supported in 3.7 and belowBugfix:       Handle uninitialized cache objectFeature: #897 Pin numpy 1.18.4Feature: #897 Pin pandas 1.0.3Bugfix:  #881 Fix for issue #815Bugfix:  #875 Fix flake8 errors1.79.4 (2020-12-01)Bugfix: Update code to work with latest version of pandasBugfix: Fix missing column handling in numpy serializer1.79.3 (2019-12-6)Bugfix: #832 Bytes to unicode for pd.SeriesFeature: #825 Max BSON size configurableBugfix: #823 Don't retry bulk write errorsFeature: #802 kwargs passed through to iterators in chunkstore1.79.2 (2019-05-21)Feature: Allow list_library caching to be tunable via a settings collection.Bugfix: #777: Handle empty columns in dataframes1.79.1 (2019-05-03)Bugfix: Pandas convert multiindexes to unicode if flag set.1.79 (2019-05-02)Bugfix: #765 Pandas data columns require encoding to convert to unicode.1.78 (2019-05-02)Bugfix: Use the default encoding (ascii) instead of bytes for unpickling.1.77 (2019-05-01)Bugfix: pypandoc not rendering README correctly for PYPIBugfix: #744 get_info on an empty dataframe raises an exceptionBugfix: #751 Handle unpickling py2 datetime and other object types with latin1Bugfix: #598 Add an option to convert py2 str to py3 str (unicode type) on read.1.74 (2019-02-28)Bugfix: #712 Pandas deprecation warning in chunkstore serializerBugFix: #670 Lots of pycodestyle fixesBugFix: #691 Fix arrays_to_mgr for pandas 0.23.4+BugFix: #698 Fix no handler in logging warningBugFix: #705 Use sort_index instead of sortlevel_BugFix: #710: Initialize SEGMENT_SHA in versions for writes and appendsFeature: #669 Experimental initial implementation of async arcticFeature: #704 Do not enable sharding by default in BSONStore.Feature: #688 Remove Chunkstore check for issue #422Feature: #717 Add a best effort cache for list_librariesBugfix: #722 Exception while reading a column from chunkstore with a column missing in the chunk1.73 (2018-11-27)Bugfix: #658 Write/append errors for Panel objects from older pandas versionsFeature: #653 Add version meta-info in arctic moduleFeature: #663 Include arctic numerical version in the metadata of the version documentFeature: #650 Implemented forward pointers for chunks in VersionStore (modes: enabled/disabled/hybrid)1.72 (2018-11-06)Feature: #577 Added implementation for incremental serializer for numpy recordsBugfix: #648 Fix issue with Timezone aware Pandas types, which don't contain hasobject attribute1.71  (2018-11-05)Bugfix: #645 Fix write errors for Pandas DataFrame that has mixed object/string types in multi-index column1.70 (2018-10-30)Bugfix: #157 Assure that serialized dataframes remain value-equivalent (e.g. avoid NaN --> 'nan' in mixed string columns)Bugfix: #608 Ensure Arctic performs well with MongoDB 3.6 (sorting)Bugfix: #629 Column kwarg no longer modifiedBugfix: #641 DateRange.intersection open/closed range fixFeature: #493 Can pass kwargs when calling MongoClient, e.g. for sslFeature: #590 Faster write handler selection for DataFrames with objectsFeature: #604 Improved handling handling for pickling serialization decidions1.69 (2018-09-12)Docs: VersionStore documentationBugfix: Issue #612 ThreadPool should be created by process using itFeature: Upsert option on appends in ChunkStore1.68 (2018-08-17)Feature: #553 Compatibility with both the new and old LZ4 APIFeature: #571 Removed the Cython LZ4 code, use the latest python-lz4Feature: #557 Threadpool based compression. Speed imrpovement and tuning benchmarks.Bugfix: fix tickstore unicode handling, support both unicode and utf-8 arraysBugfix: #591 Fix tickstore reads not returning index with localized timezoneFeature: #595 add host attribute to VersionedItem.Bugfix: #594 Enable sharding on chunkstore1.67.1 (2018-07-11)Bugfix: #579 Fix symbol corruption due to restore_version and appendBugfix: #584 Fix list_versions for a snapshot after deleting symbols in later versions1.67 (2018-05-24)Bugfix: #561 Fix PickleStore read corruption after write_metadata1.66 (2018-05-21)Bugfix: #168 Do not allow empty string as a column nameBugfix: #483 Remove potential floating point error from datetime_to_msBugfix: #271 Log when library doesnt exist on deleteFeature: MetaDataStore: added list_symbols with regexp, as_of and metadata fields matching filtersFeature: Support for serialization of DataFrames in Pandas 0.23.x1.65 (2018-04-16)Bugfix: #534 VersionStore: overwriting a symbol with different dtype (but same data format) does not
raise exceptions anymoreBugfix: #531 arctic_prune_versions: clean broken snapshot references before pruningBugfix: setup.py develop in a conda environment on MacFeature: #490 add support to numpy 1.141.63 (2018-04-06)Bugfix: #521 Clang 6.0 compiler support on macOSFeature: #510 VersionStore: support multi column in pandas DataFrames1.62 (2018-3-14)Bugfix: #517 VersionStore: append does not duplicate data in certain corner casesBugfix: #519 VersionStore: list_symbols speed improvement and fix for memory limit exceed1.61 (2018-3-2)Feature: #288 Mapping reads and writes over chunks in chunkstoreBugfix: #508 VersionStore: list_symbols and read now always returns latest versionBugfix: #512 Improved performance for list_versionsBugfix: #515 VersionStore: _prune_previous_versions now retries the cleanup operation1.60 (2018-2-13)Bugfix: #503 ChunkStore: speedup check for -1 segmentsFeature: #504 Increasing number of libraries in Arctic to 5000.1.59 (2018-2-6)Bugfix: Increase performance of invalid segment check in chunkstoreBugfix: #501 Fix the spurious data integrity exceptions at write path, due to moving chunks form the balancer1.58 (2018-1-15)Bugfix: #491 roll back the use of frombuffer to fromstring, fixes the read-only ndarray issue1.57 (2018-1-11)Feature: #206 String support for tickstoreBugfix: #486 improve mongo_retry robustness with failures for version store write/append1.56 (2017-12-21)Bugfix: #468 Re-adding compatibility for pandas 0.20.xBugfix: #476 Ensure we re-auth when a new MongoClient is created after fork1.55 (2017-12-14)Bugfix:  #439 fix cursor timeouts in chunkstore iteratorBugfix:  #450 fix error in chunkstore delete when chunk range produces empty dfBugfix:  #442 fix incorrect segment values in multi segment chunks in chunkstoreFeature: #457 enchances fix for #442 via segment_id_repair toolBugfix:  #385 exceptions during quota statistics no longer kill a writeFeature: PR#161 TickStore.max_date now returns a datetime in the 'local' timezoneFeature: #425 user defined metadata for tickstoreFeature: #464 performance improvement by avoiding unnecessary re-authenticationBugfix:  #250 Added multiprocessing safety, check for initialized MongoClient after fork.Feature: #465 Added fast operations for write only metadata and restore symbol to a version1.54 (2017-10-18)Bugfix:  #440 Fix read empty MultiIndex+tz Series1.53 (2017-10-06)Perf:    #408 Improve memory performance of version store's serializerBugfix   #394 Multi symbol read in chunkstoreBugfix:  #407 Fix segment issue on appends in chunkstoreBugfix:  Inconsistent returns on MetadataStore.appendBugfix:  #412 pandas deprecation and #289 improve exception report in numpy record serializerBugfix:  #420 chunkstore ignoring open interval date rangesBugfix:  #427 chunkstore metadata not being correctly replaced during symbol overwriteBugfix:  #431 chunkstore iterators do not handle multi segment chunks correctly1.51 (2017-08-21)Bugfix:  #397 Remove calls to deprecated methods in pymongoBugfix:  #402 Append to empty DF fails in VersionStore1.50 (2017-08-18)Feature: #396 MetadataStore.read now supports as_of argumentBugfix:  #397 Pin pymongo==3.4.01.49 (2017-08-02)Feature: #392 MetadataStoreBugfix:  #384 sentinels missing time data on chunk start/ends in ChunkStoreBugfix:  #382 Remove dependency on cython being pre-installedBugfix:  #343 Renaming libraries/collections within a namespace/database1.48 (2017-06-26)BugFix: Rollback #363, as it breaks multi-index dataframeBugfix:  #372 OSX build improvements1.47 (2017-06-19)Feature: Re-introduce #363concatflag, essentially undo-ing 1.45BugFix: #377 Fix brokenreplace_oneon BSONStore and addbulk_write1.46 (2017-06-13)Feature: #374 Shard BSONStore on_idrather thansymbol1.45 (2017-06-09)BugFix: Rollback #363, which can cause ordering issues on append1.44 (2017-06-08)Feature: #364 Expose compressHC from internal arctic LZ4 and remove external LZ4 dependencyFeature: #363 Appending older data (compare to what's exist in library) will raise. Useconcat=Trueto append only the
new bitsFeature: #371 Expose more functionality in BSONStore1.43 (2017-05-30)Bugfix:  #350 remove deprecated pandas callsBugfix:  #360 version incorrect in empty append in VersionStoreFeature: #365 add generic BSON store1.42 (2017-05-12)Bugfix: #346 fixed daterange subsetting error on very large dateframes in version storeBugfix: #351 $size queries can't use indexes, use alternative queries1.41 (2017-04-20)Bugfix: #334 Chunk range param with pandas object fails in chunkstore.get_chunk_rangesBugfix: #339 Depending on lz4<=0.8.2 to fix build errorsBugfix: #342 fixed compilation errors on Mac OSXBugfix: #344 fixed data corruption problem with concurrent appends1.40 (2017-03-03)BugFix: #330 Make Arctic._lock reentrant1.39 (2017-03-03)Feature:  #329 Add reset() method to Arctic1.38 (2017-02-22)Bugfix:  #324 Datetime indexes must be sorted in chunkstoreFeature: #290 improve performance of tickstore column reads1.37 (2017-1-31)Bugfix:  #300 to_datetime deprecated in pandas, use to_pydatetime insteadBugfix:  #309 formatting change for DateRange__str__Feature: #313 set and read user specified metadata in chunkstoreFeature: #319 Audit log support in ChunkStorBugfix:  #216 Tickstore write fails with named index column1.36 (2016-12-13)Feature: Default to hashed based shardingBugfix: retry socket errors during VersionStore snapshot operations1.35 (2016-11-29)Bugfix:  #296 Cannot compress/decompress empty string1.34 (2016-11-29)Feature: #294 Move per-chunk metadata for chunkstore to a separate collectionBugfix:  #292 Account for metadata size during size chunking in ChunkStoreFeature: #283 Support for all pandas frequency strings in ChunkStore DateChunkerFeature: #286 Add has_symbol to ChunkStore and support for partial symbol matching in list_symbols1.33 (2016-11-07)Feature:    #275 Tuple range object support in DateChunkerBugfix:     #273 Duplicate columns breaking serializerFeature:    #267 Tickstore.delete returns deleted dataDependency: #266 Remove pytest-dbfixtures in favor of pytest-server-fixtures1.32 (2016-10-25)Feature: #260 quota support on ChunkstoreBugfix: #259 prevent write of unnamed columns/indexesBugfix: #252 pandas 0.19.0 compatibility fixesBugfix: #249 open ended range reads on data without index failBugfix: #262 VersionStore.append must check data is written correctly during repackBugfix: #263 Quota: Improve the error message when near soft-quota limitPerf:   #265 VersionStore.write / append don't aggressively add indexes on each write1.31 (2016-09-29)Bugfix: #247 segmentation read fix in chunkstoreFeature: #243 add get_library_type methodBugfix: more cython changes to handle LZ4 errors properlyFeature: #239 improve chunkstore's get_info method1.30 (2016-09-26)Feature: #235 method to return chunk ranges on a symbol in ChunkStoreFeature: #234 Iterator access to ChunkStoreBugfix: #236 Cython not handling errors from LZ4 function calls1.29 (2016-09-20)Bugfix: #228 Mongo fail-over during append can leave a Version in an inconsistent stateFeature: #193 Support for different Chunkers and Serializers by symbol in ChunkStoreFeature: #220 Raise exception if older version of arctic attempts to read unsupported pickled dataFeature: #219 and #220 Support for pickling large data (>2GB)Feature: #204 Add support for library renamingFeature: #209 Upsert capability in ChunkStore's update methodFeature: #207 Support DatetimeIndexes in DateRange chunkerBugfix:  #232 Don't raise during VersionStore #append(...) if the previous append failed1.28 (2016-08-16)Bugfix: #195 Top level tickstore write with list of dicts now works with timezone aware datetimes1.27 (2016-08-05)Bugfix: #187 Compatibility with latest version of pytest-dbfixturesFeature: #182 Improve ChunkStore read/write performanceFeature: #162 Rename API for ChunkStoreFeature: #186 chunk_range on updateBugfix: #189 range delete does not update symbol metadata1.26 (2016-07-20)Bugfix: Faster TickStore querying for multiple symbols simultaneouslyBugfix: TickStore.read now respectsallow_secondary=TrueBugfix: #147 Add get_info method to ChunkStoreBugfix: Periodically re-cache the library.quota to pick up any changesBugfix: #166 Add index on SHA for ChunkStoreBugfix: #169 Dtype mismatch in chunkstore updatesFeature: #171 allow deleting of values within a date range in ChunkStoreBugfix: #172 Fix date range bug when querying dates in the middle of chunksBugfix: #176 Fix overwrite failures in ChunkstoreBugfix: #178 - Change how start/end dates are populated in the DB, also fix append so it works as expected.Bugfix: #43 - Remove dependency on hardcoded Linux timezone files1.25 (2016-05-23)Bugfix: Ensure that Tickstore.write doesn't allow out of order messagesBugfix: VersionStore.write now allows writing 'None' as a value1.24 (2016-05-10)Bugfix: Backwards compatibility reading/writing documents with previous versions of Arctic1.22 (2016-05-09)Bugfix: #109 Ensure stable sort during Arctic readFeature: New benchmark suite using ASVBugfix: #129 Fixed an issue where some chunks could get skipped during a multiple-symbol TickStore readBugfix: #135 Fix issue with different datatype returned from pymongo in python3Feature: #130 New Chunkstore storage type1.21 (2016-03-08)Bugfix: #106 Fix Pandas Panel storage for panels with different dimensions1.20 (2016-02-03)Feature: #98 Add initial_image as optional parameter on tickstore write()Bugfix: #100 Write error on end field when writing with pandas dataframes1.19 (2016-01-29)Feature: Add python 3.3/3.4 supportBugfix: #95 Fix raising NoDataFoundException across multiple low level libraries1.18 (2016-01-05)Bugfix: #81 Fix broken read of multi-index DataFrame written by old version of ArcticBugfix: #49 Fix strifying tickstore1.17 (2015-12-24)Feature: Add timezone suppport to store multi-index dataframesBugfix:  Fixed broken sdist releases1.16 (2015-12-15)Feature: ArticTransaction now supports non-audited 'transactions':audit=Falsewith ArcticTransaction(Arctic('hostname')['some_library'], 'symbol', audit=False) as at:
...This is useful for batch jobs which read-modify-write and don't want to clash with
 concurrent writers, and which don't require keeping all versions of a symbol.1.15 (2015-11-25)Feature: get_info API added to version_store.1.14 (2015-11-25)1.12 (2015-11-12)Bugfix: correct version detection for Pandas >= 0.18.Bugfix: retrying connection initialisation in case of an AutoReconnect failure.1.11 (2015-10-29)Bugfix: Improve performance of saving multi-index Pandas DataFrames
by 9xBugfix: authenticate should propagate non-OperationFailure exceptions
(e.g. ConnectionFailure) as this might be indicative of socket failuresBugfix: return 'deleted' state in VersionStore.list_versions() so that
callers can pick up on the head version being the delete-sentinel.1.10 (2015-10-28)Bugfix: VersionStore.read(date_range=...) could do the wrong thing with
TimeZones (which aren't yet supported for date_range slicing.).1.9 (2015-10-06)Bugfix: fix authentication race condition when sharing an Arctic
instance between multiple threads.1.8 (2015-09-29)Bugfix: compatibility with both 3.0 and pre-3.0 MongoDB for
querying current authentications1.7 (2015-09-18)Feature: Add support for reading a subset of a pandas DataFrame
in VersionStore.read by passing in an arctic.date.DateRangeBugfix: Reauth against admin if not auth'd against a library a
specific library's DB.  Sometimes we appear to miss admin DB auths.
This is to workaround that until we work out what the issue is.1.6 (2015-09-16)Feature: Add support for multi-index Bitemporal DataFrame storage.
This allows persisting data and changes within the DataFrame making it
easier to see how old data has been revised over time.Bugfix: Ensure we call the error logging hook when exceptions occur1.5 (2015-09-02)Always use the primary cluster node for 'has_symbol()', it's safer1.4 (2015-08-19)Bugfixes for timezone handling, now ensures use of non-naive datetimesBugfix for tickstore read missing images1.3 (2015-08-011)Improvements to command-line control scripts for users and librariesBugfix for pickling top-level Arctic object1.2 (2015-06-29)Allow snapshotting a range of versions in the VersionStore, and
snapshot all versions by default.1.1 (2015-06-16)Bugfix for backwards-compatible unpickling of bson-encoded dataAdded switch for enabling parallel lz4 compression1.0 (2015-06-14)Initial public release