Package: dpsailab-util
DPS_UtilThis repository contain all util.Install:Get athttps://pypi.org/project/dpsailab-util/pip install dpsailab-utilChange log:Ver 1.0.0:Compression -> DONEKafkaWrapper -> DONERedisWrapper -> DONEVector -> DONEMedia -> DONEComputer Vision -> DONEDistance -> DONEHashing -> DONEVer 1.0.1:Added image_info -> Get (format, width, height) of image without decoding.Upgrade imdecode -> Auto decode image but need not provide encode_type.Ver 1.0.2:Change 'numpy.pool' to 'vector.pool'Fix bugsVer 1.0.3:Compression:Support list compression:compress_list, decompres_listDistance:Change distance.function -> vector.distanceVer 1.1.0:Added AttrDict, FixedDict, FixedTypeDictVer 1.1.1-1.1.7:Fix bugsAdded UniqueTypeDictFix bugsChanged module fixdict -> defaultdictChanged FixedDict -> DefaultDictChanged FixedTypeDict -> DefaultTypeDictFix bugs of attrdict.AttrDict, attrdict.UniqueTypeDict, attrdict.DefaultDictFix bugs of compress_list, decompress_listVer 1.1.8:Added EnvironmentVer 1.1.9-1.1.18:Added Environment.to_lower -> Useful when extracting to kwarg.Fix bugs attrdictChanged hashfunc default from sha1 -> md5Supportcallof DefaultDict, which update data from buffer.Fix bugs EnvironmentVer 1.1.19:Added pre-define output array of decompress_ndarraySupport add new key environment.Ver 1.1.20-1.1.21:Edit loggerFix bugs vector.poolChanged VectorPool base on numpy.mmap -> VectorPoolMMapAdded VectorPool base on numpy.ndarrayRestructure VectorPoolBase -> Speed up, fix bugsAdded VectorPoolBase.insertVer 1.1.22:Add document of compression, media module.Fix bugs imageChange media.image.scale -> media.image.zoomVer 1.1.23-24:Fix bugs logFix bugs VectorPoolMMap.MIN_SIZEVer 1.1.25-26:Fix bugs attrdictAdded add, remove method attrdictRename variable of pool.VectorPoolBaseVer 1.1.27:Change namespace attrdict.UniqueTypeDict -> attrdict.TypedDictRestructure attdict.AttrDict, attdict.DefaultDict, attrdict.TypedDictSupport annotations aliasSpeed up attrdictSupport initial attrdict with iterable, generator.Support to_buffer method with optimize compression.Support recreate dict with from_buffer.Lightweight than pickle.dumpsVer 1.2.0:CV:Added Face AlignerTransforms image methodsmedia.image:Added scale, crop_center, flipFix bugsmedia.video: support FPS report.triangle (new)Ver 1.3.0:Support decorator attrdictSupport decorator environmentImplemented Confluent-Kafka with:Iterator methodClose event.Auto_commitOptimize configs for ThroughOutFix bug KafkaErrorTodo:Implement Configs librdkafkaSort -> implement natsort -https://github.com/SethMMorton/natsortFeaturesCompression Lossless:Support type: ndnumpy, bytesCompress by blosc. It support multi compressor and multi-thread.KafkaWrapper:Wrapping Consumer and Producer with default setting and security.RedisWrapperWrapping Redis Connector with default setting and security.Vector Pool:VectorPoolBasehandlenumpy.ndarraythat save a lot of RAM by re-use existed memory space.Easier thannumpy.ndarray. Likelylist.Reduce RAMHigh Speed IOAuto ScaleSupport backup and recovery data in a second.Media:ImplementedOpenCVwith:TurboJPEG (https://github.com/lilohuang/PyTurboJPEG)FFmpeg(https://github.com/kkroening/ffmpeg-python)To: improve read & write (image, video) IO speed.Faster thanOpenCV:2-6x with JPEG Encoder1.1x with others.Added some function which used frequently.More info: find in dpsutil.mediaComputer Vision (cv):Added Face Align with five landmark.Distance:Added cosine_similarityAdded cosineAdded euclidean_distanceAdded convert distance functions*Note: all function execute in numpy.Attributes dict:AttrDict: will help you get value of key via attributes. Implement attrdict.AttrDictDefaultDict: help cover your dict with (keys, values) that was defined before. Implement from AttrDictTypedDict: help cover your dict when set item. Implement from AttrDict.DefaultTypeDict: dict only access one type for all element.Support initial attrdict with iterable, generator.Support to_buffer method with optimize compression.Support recreate dict with from_buffer.Lightweight than pickle.dumpsEnvironement:Environment: Auto pair environment parameters fit with default, which provided before.Other:HashingSortIssue:Cmake error during install bloscFirstly, you need install scikit$ pip install scikit-buildAfter that, follow instuction to install Cmake:https://cliutils.gitlab.io/modern-cmake/chapters/intro/installing.htmlPython$ pip install cmake>=3.12Not found 'FFmpeg':Find FFmppeg lib at:https://www.ffmpeg.org/download.htmlLinux:sudo apt-get install ffmpegNot found 'libjpeg-turbo':Find FFmppeg lib at:https://libjpeg-turbo.org/Documentation/OfficialBinariesLinux:$ sudo apt install libturbojpegNot found Redis or Kafka server:Make sure your Redis or Kafka server started.Make sure correct username & password.Not found 'Could NOT find OpenSSL':SSL development libraries have to be installedhttps://github.com/openssl/openssl#build-and-installUbuntu:$ apt-get install libssl-dev libffi-devDevelop by: TinDang
Maintain by: CanhTran
OS: Ubuntu Bionic 18.04Python: 3.6.9 stable