Package: preprocessing-pgp
preprocessing_pgppreprocessing_pgp-- The Preprocessing library for any kind of data -- is a suit ofopen source Python modules, preprocessing techniquessupporting research and development in Machine Learning. preprocessing_pgp requires Python version3.6, 3.7, 3.8, 3.9, 3.10InstallationTo install thecurrent release:pipinstallpreprocessing-pgpTo install the release withspecific version(e.g. 0.1.3):pipinstallpreprocessing-pgp==0.2.9To upgrade package tolatest version:pipinstall--upgradepreprocessing-pgpFeatures1. Vietnamese Naming Functions1.1. Preprocessing Namespython>>>frompreprocessing_pgp.name.preprocessimportbasic_preprocess_name>>>basic_preprocess_name('Phan Thị    Thúy    Hằng *$%!@#')PhanThịThúyHằng1.2. Enrich Vietnamese Names (Pending...)python>>>importpandasaspd>>>frompreprocessing_pgp.name.enrich_nameimportprocess_enrich>>>data=pd.read_parquet('/path/to/data.parquet')>>>enrich_data=process_enrich(data,name_col='name')CleansingTakes0m0sEnrichnamestakes5m10s>>>enrich_data.columnsIndex(['name','predict','final'],dtype='object')1.3. Extract customer type from name (New Feature)In big data platform, user might enter not just there name into the name field but many others.This module currently support detection of following type:customer: The name of thecustomercompany: The name of anycompany relatedbiz: The name of anybusiness relatededu: The name of any type ofeducation relatedmedical: The name of anymedical relatedpython>>>importpandasaspd>>>frompreprocessing_pgp.name.type.extractorimportprocess_extract_name_type>>>data=pd.read_parquet('/path/to/data.parquet')>>>extracted_data=process_extract_name_type(data,name_col='name')Cleansingnamestakes0m0sFormattingnamestakes0m0sExtractingcustomer's type takes 0m0s>>>extracted_data.columnsIndex(['username','customer_type'],dtype='object')2. Extracting Vietnamese Phonespython>>>importpandasaspd>>>frompreprocessing_pgp.phone.extractorimportprocess_convert_phone>>>data=pd.read_parquet('/path/to/data.parquet')>>>extracted_data=process_convert_phone(data,phone_col='phone')Convertingphonestakes0m1s>>>extracted_data.columnsIndex(['phone','is_phone_valid','is_mobi','is_new_mobi','is_old_mobi','is_new_landline','is_old_landline','phone_convert','phone_vendor','tail_phone_type'],dtype='object')3. Verify Vietnamese Card IDspython>>>importpandasaspd>>>frompreprocessing_pgp.card.validationimportprocess_verify_card>>>data=pd.read_parquet('/path/to/data.parquet')>>>verified_data=process_verify_card(data,card_col='card_id')Processcleaningcardid...Verifyingcardidtakes0m3s>>>verified_data.columnsIndex(['card_id','clean_card_id','is_valid','is_personal_id','is_passport','is_driver_license'],dtype='object')4. Extract Information in Vietnamese AddressAll the region codes traced are retrieve fromĐơn Vị Hành Chính Việt NamApart from original columns ofdataframe, we also generate columns with specific meanings:cleaned_<address_col>: Thecleaned addressretrieve from the raw address columnlevel 1: The raw city extracted from thecleaned addressbest level 1: Thebeautified citytraced from extracted raw citylevel 1 code: The generatedcity codelevel 2: The raw district extracted from thecleaned addressbest level 2: Thebeautified districttraced from extracted raw districtlevel 2 code: The generateddistrict codelevel 3: The raw ward extracted from thecleaned addressbest level 3: Thebeautified wardtraced from extracted raw wardlevel 3 code: The generatedward coderemained address: Theremaining addressnot being extractedpython>>>importpandasaspd>>>frompreprocessing_pgp.address.extractorimportextract_vi_address>>>data=pd.read_parquet('/path/to/data.parquet')>>>extracted_data=extract_vi_address(data,address_col='address')Cleansingtakes0m0sExtractingtakes0m22sCodegenerationtakes0m3s>>>extracted_data.columnsIndex(['address','cleaned_address','level 1','best level 1','level 2','best level 2','level 3','best level 3','remained address','level 1 code','level 2 code','level 3 code'],dtype='object')5. Validate email addressA valid email is consist of:Large company email's address (@gmail, @yahoo, @outlook, etc.)Common email address (contains at least a alphabet character in email's name)Education email (can start with a number)Not auto-emailApart from original columns ofdataframe, we also generate columns with specific meanings:is_email_valid: indicator of whether the email is valid or notpython>>>importpandasaspd>>>frompreprocessing_pgp.email.validatorimportprocess_validate_email>>>data=pd.read_parquet('/path/to/data.parquet')>>>validated_data=process_validate_email(data,email_col='email')Cleansingemailtakes0m0sValidatingemailtakes0m22s