Package: cstl
CSTL : The C++ Standard Template Library (STL) for PythonIn thiscstltool, we wrap several C++ STL containers to use in Python. The containers use native C++ implementation and does not have theCopy-on-Write issuelike nativelist, anddictin Python (Someone refers to it as memory leakage in multiprocessing which always happens in all Python native objects with ref count). Though it is designed to solve the CoW issue, it can also be used in scenarios where a standard C++ container is needed.InstallInstall frompip:pip install cstlBuild from source:conda install swig # You should first install swig
git clone https://github.com/fuzihaofzh/cstl.git
cd cstl
./build.sh
python setup.py install --single-version-externally-managed --record files.txtUsageimportcstl# Directly covert containers from pythonv=cstl.frompy({"1":[1,2,3],"2":[4,5,6]})v["1"][2]=10pv=cstl.topy(v)print(pv)# You can also explictly specify the typevec=cstl.VecInt([1,2,3,4,5,6,7])print(vec[2])#3vec[2]=1print(vec[2])#1vec.append(10)print(vec[-1])#10# User should explictly convert std::vector into list as follows:print(list(vec))#[1, 2, 1, 4, 5, 6, 7, 10]vmif=cstl.VecMapIntFloat([{1:3.4},{4:5.5}])print(vmif[0][1])#3.4000000953674316Please refer to more usage intest/test/py.Supported DatatypeWe support the following data types as the elements in the containersPython TypeC++ TypeCan be dict keyintintYesintstd::int64Yesstrstd::stringYesfloatfloatNodoubledoubleNoboolboolNoSupported ContainersThe supported containers are listed as followsPython StructureC++ Containerliststd::vectordictstd::unordered_mapsetstd::unordered_setWe also support nested container, namely, structure likestd::unordered_map< std::string,std::vector< bool > >is supported. Currently, at most 2 nested layers are supported. If you want to support more layers, simply uncomment the linefull.update(third)ingenerate_swig.pyand compile from the source. But please note the the generated files could be very large.Please be noted that if you want to pass a Python set to set incstl, you should first convert it into a list. This is aknown issuefor swig, let's see whether it will be resolved soon.Speed Comparisonpythonnumpycstlpytorchadd10.190.280.9114.714read0.1610.20.5261.033sliceread0.3270.2640.6831.381append0.204>100.351>10pop>10>100.595>10It can be concluded from the table that cstl is slower than python native list and numpy in basic tasks. However, it is faster in some specific task. Most importantly, it sovles the CoW issue and provide more data structures than numpy.Copy-on-Write Issue in PythonCopy-on-writeis a feature and not a bug for Python. Usually, in python multiprocessing, if we have a large data shared by each process, we will see as the program runs, the engaged memory grows gradually and finally occupy all the machine's memory and raise a Memory Error. Someone refers to it as memory leaky in the multiprocess. However, this is caused by a feature of Python. It is because, in multi-processing programs, the shared object will be copied to each process if they access the data. However, if the data is large and we use several processes, the memory cannot hold a separate copy for each process. This cannot be solved in Python as all Python's native structures with ref count have such problems (feature?). A more detailed discussion can be found athttps://github.com/pytorch/pytorch/issues/13246. Many other containers like pytorch, numpy. However, they do not support data structure like nested map.Supported STL Containers ListWe support the following nested containers. If you need more than 2 layers of nested containers please refer toSupported Containerscstl nameC++ classVecIntstd::vector<int>VecStrstd::vector<std::string>VecFloatstd::vector<float>VecDoublestd::vector<double>VecBoolstd::vector<bool>VecLongstd::vector<std::int64_t>SetIntstd::unordered_set<int>SetStrstd::unordered_set<std::string>SetLongstd::unordered_set<std::int64_t>MapIntIntstd::unordered_map<int, int>MapIntStrstd::unordered_map<int, std::string>MapIntFloatstd::unordered_map<int, float>MapIntDoublestd::unordered_map<int, double>MapIntBoolstd::unordered_map<int, bool>MapIntLongstd::unordered_map<int, std::int64_t>MapStrIntstd::unordered_map<std::string, int>MapStrStrstd::unordered_map<std::string, std::string>MapStrFloatstd::unordered_map<std::string, float>MapStrDoublestd::unordered_map<std::string, double>MapStrBoolstd::unordered_map<std::string, bool>MapStrLongstd::unordered_map<std::string, std::int64_t>MapLongIntstd::unordered_map<std::int64_t, int>MapLongStrstd::unordered_map<std::int64_t, std::string>MapLongFloatstd::unordered_map<std::int64_t, float>MapLongDoublestd::unordered_map<std::int64_t, double>MapLongBoolstd::unordered_map<std::int64_t, bool>MapLongLongstd::unordered_map<std::int64_t, std::int64_t>VecVecIntstd::vector<std::vector<int> >VecVecStrstd::vector<std::vector<std::string> >VecVecFloatstd::vector<std::vector<float> >VecVecDoublestd::vector<std::vector<double> >VecVecBoolstd::vector<std::vector<bool> >VecVecLongstd::vector<std::vector<std::int64_t> >VecSetIntstd::vector<std::unordered_set<int> >VecSetStrstd::vector<std::unordered_set<std::string> >VecSetLongstd::vector<std::unordered_set<std::int64_t> >VecMapIntIntstd::vector<std::unordered_map<int, int> >VecMapIntStrstd::vector<std::unordered_map<int, std::string> >VecMapIntFloatstd::vector<std::unordered_map<int, float> >VecMapIntDoublestd::vector<std::unordered_map<int, double> >VecMapIntBoolstd::vector<std::unordered_map<int, bool> >VecMapIntLongstd::vector<std::unordered_map<int, std::int64_t> >VecMapStrIntstd::vector<std::unordered_map<std::string, int> >VecMapStrStrstd::vector<std::unordered_map<std::string, std::string> >VecMapStrFloatstd::vector<std::unordered_map<std::string, float> >VecMapStrDoublestd::vector<std::unordered_map<std::string, double> >VecMapStrBoolstd::vector<std::unordered_map<std::string, bool> >VecMapStrLongstd::vector<std::unordered_map<std::string, std::int64_t> >VecMapLongIntstd::vector<std::unordered_map<std::int64_t, int> >VecMapLongStrstd::vector<std::unordered_map<std::int64_t, std::string> >VecMapLongFloatstd::vector<std::unordered_map<std::int64_t, float> >VecMapLongDoublestd::vector<std::unordered_map<std::int64_t, double> >VecMapLongBoolstd::vector<std::unordered_map<std::int64_t, bool> >VecMapLongLongstd::vector<std::unordered_map<std::int64_t, std::int64_t> >MapIntVecIntstd::unordered_map<int, std::vector<int> >MapIntVecStrstd::unordered_map<int, std::vector<std::string> >MapIntVecFloatstd::unordered_map<int, std::vector<float> >MapIntVecDoublestd::unordered_map<int, std::vector<double> >MapIntVecBoolstd::unordered_map<int, std::vector<bool> >MapIntVecLongstd::unordered_map<int, std::vector<std::int64_t> >MapIntSetIntstd::unordered_map<int, std::unordered_set<int> >MapIntSetStrstd::unordered_map<int, std::unordered_set<std::string> >MapIntSetLongstd::unordered_map<int, std::unordered_set<std::int64_t> >MapIntMapIntIntstd::unordered_map<int, std::unordered_map<int, int> >MapIntMapIntStrstd::unordered_map<int, std::unordered_map<int, std::string> >MapIntMapIntFloatstd::unordered_map<int, std::unordered_map<int, float> >MapIntMapIntDoublestd::unordered_map<int, std::unordered_map<int, double> >MapIntMapIntBoolstd::unordered_map<int, std::unordered_map<int, bool> >MapIntMapIntLongstd::unordered_map<int, std::unordered_map<int, std::int64_t> >MapIntMapStrIntstd::unordered_map<int, std::unordered_map<std::string, int> >MapIntMapStrStrstd::unordered_map<int, std::unordered_map<std::string, std::string> >MapIntMapStrFloatstd::unordered_map<int, std::unordered_map<std::string, float> >MapIntMapStrDoublestd::unordered_map<int, std::unordered_map<std::string, double> >MapIntMapStrBoolstd::unordered_map<int, std::unordered_map<std::string, bool> >MapIntMapStrLongstd::unordered_map<int, std::unordered_map<std::string, std::int64_t> >MapIntMapLongIntstd::unordered_map<int, std::unordered_map<std::int64_t, int> >MapIntMapLongStrstd::unordered_map<int, std::unordered_map<std::int64_t, std::string> >MapIntMapLongFloatstd::unordered_map<int, std::unordered_map<std::int64_t, float> >MapIntMapLongDoublestd::unordered_map<int, std::unordered_map<std::int64_t, double> >MapIntMapLongBoolstd::unordered_map<int, std::unordered_map<std::int64_t, bool> >MapIntMapLongLongstd::unordered_map<int, std::unordered_map<std::int64_t, std::int64_t> >MapStrVecIntstd::unordered_map<std::string, std::vector<int> >MapStrVecStrstd::unordered_map<std::string, std::vector<std::string> >MapStrVecFloatstd::unordered_map<std::string, std::vector<float> >MapStrVecDoublestd::unordered_map<std::string, std::vector<double> >MapStrVecBoolstd::unordered_map<std::string, std::vector<bool> >MapStrVecLongstd::unordered_map<std::string, std::vector<std::int64_t> >MapStrSetIntstd::unordered_map<std::string, std::unordered_set<int> >MapStrSetStrstd::unordered_map<std::string, std::unordered_set<std::string> >MapStrSetLongstd::unordered_map<std::string, std::unordered_set<std::int64_t> >MapStrMapIntIntstd::unordered_map<std::string, std::unordered_map<int, int> >MapStrMapIntStrstd::unordered_map<std::string, std::unordered_map<int, std::string> >MapStrMapIntFloatstd::unordered_map<std::string, std::unordered_map<int, float> >MapStrMapIntDoublestd::unordered_map<std::string, std::unordered_map<int, double> >MapStrMapIntBoolstd::unordered_map<std::string, std::unordered_map<int, bool> >MapStrMapIntLongstd::unordered_map<std::string, std::unordered_map<int, std::int64_t> >MapStrMapStrIntstd::unordered_map<std::string, std::unordered_map<std::string, int> >MapStrMapStrStrstd::unordered_map<std::string, std::unordered_map<std::string, std::string> >MapStrMapStrFloatstd::unordered_map<std::string, std::unordered_map<std::string, float> >MapStrMapStrDoublestd::unordered_map<std::string, std::unordered_map<std::string, double> >MapStrMapStrBoolstd::unordered_map<std::string, std::unordered_map<std::string, bool> >MapStrMapStrLongstd::unordered_map<std::string, std::unordered_map<std::string, std::int64_t> >MapStrMapLongIntstd::unordered_map<std::string, std::unordered_map<std::int64_t, int> >MapStrMapLongStrstd::unordered_map<std::string, std::unordered_map<std::int64_t, std::string> >MapStrMapLongFloatstd::unordered_map<std::string, std::unordered_map<std::int64_t, float> >MapStrMapLongDoublestd::unordered_map<std::string, std::unordered_map<std::int64_t, double> >MapStrMapLongBoolstd::unordered_map<std::string, std::unordered_map<std::int64_t, bool> >MapStrMapLongLongstd::unordered_map<std::string, std::unordered_map<std::int64_t, std::int64_t> >MapLongVecIntstd::unordered_map<std::int64_t, std::vector<int> >MapLongVecStrstd::unordered_map<std::int64_t, std::vector<std::string> >MapLongVecFloatstd::unordered_map<std::int64_t, std::vector<float> >MapLongVecDoublestd::unordered_map<std::int64_t, std::vector<double> >MapLongVecBoolstd::unordered_map<std::int64_t, std::vector<bool> >MapLongVecLongstd::unordered_map<std::int64_t, std::vector<std::int64_t> >MapLongSetIntstd::unordered_map<std::int64_t, std::unordered_set<int> >MapLongSetStrstd::unordered_map<std::int64_t, std::unordered_set<std::string> >MapLongSetLongstd::unordered_map<std::int64_t, std::unordered_set<std::int64_t> >MapLongMapIntIntstd::unordered_map<std::int64_t, std::unordered_map<int, int> >MapLongMapIntStrstd::unordered_map<std::int64_t, std::unordered_map<int, std::string> >MapLongMapIntFloatstd::unordered_map<std::int64_t, std::unordered_map<int, float> >MapLongMapIntDoublestd::unordered_map<std::int64_t, std::unordered_map<int, double> >MapLongMapIntBoolstd::unordered_map<std::int64_t, std::unordered_map<int, bool> >MapLongMapIntLongstd::unordered_map<std::int64_t, std::unordered_map<int, std::int64_t> >MapLongMapStrIntstd::unordered_map<std::int64_t, std::unordered_map<std::string, int> >MapLongMapStrStrstd::unordered_map<std::int64_t, std::unordered_map<std::string, std::string> >MapLongMapStrFloatstd::unordered_map<std::int64_t, std::unordered_map<std::string, float> >MapLongMapStrDoublestd::unordered_map<std::int64_t, std::unordered_map<std::string, double> >MapLongMapStrBoolstd::unordered_map<std::int64_t, std::unordered_map<std::string, bool> >MapLongMapStrLongstd::unordered_map<std::int64_t, std::unordered_map<std::string, std::int64_t> >MapLongMapLongIntstd::unordered_map<std::int64_t, std::unordered_map<std::int64_t, int> >MapLongMapLongStrstd::unordered_map<std::int64_t, std::unordered_map<std::int64_t, std::string> >MapLongMapLongFloatstd::unordered_map<std::int64_t, std::unordered_map<std::int64_t, float> >MapLongMapLongDoublestd::unordered_map<std::int64_t, std::unordered_map<std::int64_t, double> >MapLongMapLongBoolstd::unordered_map<std::int64_t, std::unordered_map<std::int64_t, bool> >MapLongMapLongLongstd::unordered_map<std::int64_t, std::unordered_map<std::int64_t, std::int64_t> >