Package: cads-sdk
cads-sdk: Functions to help Data Scientist work more effectively with unstructured data and datalakeWhat is it?cads-sdkFunction to help Data Scientist work more effectively with unstructured data. Include different function work with image, audioMain FeaturesHere are just a few of the things that cads-sdk does well:Data pre-processing: Faster upto 25% compare with cv2.imread()Image pre-processing, convert from Image Folder/Zipfile to Parquet/delta, ready for trainingAudio pre-processing, convert from Folder Audio, ready for trainingOptimize storage: Does not reduce image quality but optimize memory 12%Decrease number of small fileTake advantage of compression zstd parquetView image/audio but not get system down (browser ran out of memory)InstallBinary installers for the latest released version are available at thePython
Package Index (PyPI).# with PyPIpipinstallcads-sdkDependenciesspark-sdk - PySpark, PyArrow add on functionopencv-python - Wrapper package for OpenCV python bindingspetastorm - Petastorm is a library enabling the use of Parquet storage from Tensorflow, Pytorch, and other Python-based ML training frameworkspandas - Powerful data structures for data analysis, time series, and statisticsInstallation from sourcesTo install cads-sdk from source you needCythonin addition to the normal
dependencies above. Cython can be installed from PyPI:pipinstallcythonIn thecads-sdkdirectory (same one where you found this file after
cloning the git repo), execute:pythonsetup.pyinstallDocumentationThe official documentation is hosted on PyData.org:https://pandas.pydata.org/pandas-docs/stableImageConvert a folder image to parquetfromcads_sdk.nosql.converterimportConvertFromFolderImageconverter=ConvertFromFolderImage(input_path="/path/to/folder/**/*.jpg",input_type='jpg'# 'jpg' | ('jpg', 'png')input_recursive=True,#setting outputoutput_path=f"file:/output/path/image_storage",# setting converterimage_type='jpg',image_color=3,resize_mode=None,# |padding|resizesize=[(212,212),(597,597)],)converter.execute()# convert directly from .zip file to parquetfromcads_sdk.nosql.converterimportConvertFromZipImageconverter=ConvertFromZipImage(input_path="/path/to/image_storage/ETHZ.zip",input_recursive=True,# will loop through folder to get all patterninput_type='jpg'# 'jpg' | ('jpg', 'png')#setting outputoutput_path=f"file:/output/path/img_ethz.parquet",table_name='img_ethz',database='default',file_format='parquet',# delta|parquetcompression='zstd',# zstd|snappy# setting converterimage_type='png',image_color=3,resize_mode=None,# |padding|resizesize=[(1080,1920)],debug=False)converter.execute()Convert a Image parquet file back to Image Folderfromcads_sdk.nosql.converterimportConvertToFolderImageconverter=ConvertToFolderImage(input_path='/user/username/image/img_user_device_jpg_212_212.parquet',raw_input_path="/home/username/image_storage/device_images/**/*.jpg",output_path='./abc/')converter.execute()Function to read imagefromcads.nosqlimportdisplayimportcads_sdkasssdf=ss.sql("""select * from parquet.`/user/duyvnc/image/img_images_jpg_212_212.parquet`""")pdf=df.toPandasImage(limit=50)pdfpdf=ss.sql("""select *from parquet.`file:/home/duyvnc/image_storage/img_mot17_1080_1920.parquet`limit 100""").toPandasImage(mode='BGR')Pytorch APIfromcads_sdk.nosqlimportcodecfrompetastormimportmake_reader,TransformSpecfrompetastorm.pytorchimportDataLoadernum_epochs=10withDataLoader(reader=make_reader('{}/train'.format(dataset_url),reader_pool_type='dummy',num_epochs=num_epochs,transform_spec=transform),batch_size=32)astrain_loader:train(model,device,train_loader,2000,optimizer,num_epochs)AudioSuport pcm, mp3, wav formatConvert a folder audio to parquetfromcads_sdk.nosql.converterimportConvertFromFolderImageconverter=ConvertFromFolderAudio(input_path='/path/to/audio_wav/*.wav',#(1)input_type='wav'# 'wav'| 'mp3' | 'pcm'input_recursive=False,output_path=f"file:/output/path/audio_wav.parquet",)converter.execute()Convert a parquet to folder audioconverter=ConvertToFolderAudio(input_path='file:/path/to/audio_wav.parquet',raw_input_path='/path/to/audio_wav/*.wav',#(1) # auto replace '/path/to/audio_wav/' to ''output_path='./output/path',write_mode="recovery")converter.execute()Listen audio in parquetfromcads_sdk.nosql.displayimportAudioAudio('file:/path/to/audio_mp3.parquet')Videofromcads_sdk.nosql.converterimportConvertFromVideo2Imageconverter=ConvertFromVideo2Image(input_path='/home/username/vid/palawan1.mp4',output_path=f"file:/home/username/vid_image.parquet",)converter.execute()For more information use class instanceConvertFromFolderImage.__doc__