Package: skt
SKT PackageThis is highly site dependent package.
Resources are abstracted into package structure.UsageHive metastorefromskt.yeimportget_hmsc=get_hms()c.get_partition_names("db","table")c.close()Hash and unhashfromskt.lakeimporthash_sfromskt.lakeimportunhash_sunhashed_list=['0000000000']hashed_list=hash_s(unhashed_list)unhash_s(hashed_list)Execute hive query without fetch resultfromskt.yeimporthive_executehive_execute(ddl_or_ctas_query)Fetch resultset from hive queryfromskt.yeimporthive_get_resultresult_set=hive_get_result(select_query)Get pandas dataframe from hive qeruy resultsetfromskt.yeimporthive_to_pandaspandas_df=hive_to_pandas(hive_query)Get pandas dataframe from parquet file in hdfsfromskt.yeimportparquet_to_pandaspandas_df=parquet_to_pandas(hdfs_path)Save pandas dataframe as parquet in hdfsfromskt.yeimportget_sparkfromskt.yeimportpandas_to_parquetspark=get_spark()pandas_to_parquet(pandas_df,hdfs_path,spark)# we need spark for this operationspark.stop()Work with sparkfromskt.yeimportget_sparkspark=get_spark()# do with spark sessionspark.stop()Work with spark-bigquery-connector# SELECTfromskt.gcpimportbq_table_to_pandaspandas_df=bq_table_to_pandas("dataset","table_name",["col_1","col_2"],"2020-01-01","svc_mgmt_num is not null")# INSERTfromskt.gcpimportpandas_to_bq_tablepandas_to_bq_table(pandas_df,"dataset","table_name","2020-03-01")Send slack messagefromskt.yeimportslack_sendtext='Hello'username='airflow'channel='#leavemealone'slack_send(text=text,username=username,channel=channel)# Send dataframe as textdf=pd.DataFrame(data={'col1':[1,2],'col2':[3,4]})slack_send(text=df,username=username,channel=channel,dataframe=True)Get bigquery clientfromskt.gcpimportget_bigquery_clientbq=get_bigquery_client()bq.query(query)IPython BigQuery Magicfromskt.gcpimportimport_bigquery_ipython_magicimport_bigquery_ipython_magic()query_params={"p_1":"v_1","dataset":"mnoai",}%%bq--params$query_paramsSELECTc_1FROM{dataset}.user_logsWHEREc_1=@p_1Access MLSfromskt.mlsimportset_model_namefromskt.mlsimportget_recent_model_pathfromskt.yeimportget_pkl_from_hdfsset_model_name(COMM_DB,params)path=get_recent_model_path(COMM_DB,model_key)model=get_pkl_from_hdfs(f'{path})MLS Model Registry (Upload model_binary(model.tar.gz) / model_meta(model.json) to AWS S3 from YE)fromskt.mlsimportsave_model# model object generated by LightGBM or XGBoostmodel# model namemodel_name='sample_model'# model versionmodel_version='v1'# AWS ENV in 'stg / prd / dev' (default is 'stg')aws_env='stg'# List of features used in ML Model in string type (only for XGBoost model_type)feature_list=['feature_1','feature_2','feature_3']# Force to overwrite model files on S3 if exists (default is False)force=Falsesave_model(model,model_name,model_version,aws_env,force)MLSmeta_table&meta_table_itemrelated methodsfromskt.mlsimportget_meta_tablefromskt.mlsimportcreate_meta_table_itemfromskt.mlsimportupdate_meta_table_itemfromskt.mlsimportget_meta_table_itemfromskt.mlsimportmeta_table_to_pandasfromskt.mlsimportpandas_to_meta_table# Get a meta_table infoget_meta_table(meta_table_name,aws_env,edd)# Create a meta_itemcreate_meta_table_item(meta_table_name,item_name,item_dict,aws_env,edd)# Update a meta_itemupdate_meta_table_item(meta_table_name,item_name,item_dict,aws_env,edd)# Get a meta_itemget_meta_table_item(meta_table_name,item_name,aws_env,edd)# Get a meta_table as pandas dataframemeta_table_to_pandas(meta_table_name,aws_env,edd)# Update pandas dataframe to meta_tablepandas_to_meta_table(method,meta_table_name,dataframe,key,values,aws_env,edd)# For the detal, use ?{method} to get detailed info (ex. ?get_meta_table)# For the user of EDD, must set edd=TrueMLSmodel_metarelated methods(*Need to setuserfor the ml_model)fromskt.mlsimportget_ml_modelfromskt.mlsimportcreate_meta_table_itemfromskt.mlsimportupdate_meta_table_item# Get a ml_modelget_ml_model(user,model_name,model_version,aws_env,edd)# Get a model_meta of ml_modelget_ml_model_meta(user,model_name,model_version,aws_env,edd)# Update or Create meta_item(s)update_ml_model_meta(user,model_name,model_version,model_meta_dict,aws_env,edd)# For the detal, use ?{method} to get detailed info (ex. ?get_ml_model)# For the user of EDD, must set edd=TrueUse NES CLInes input_notebook_url -p k1 v1 -p k2 v2 -p k3 v3Use github utilfromskt.yeimportget_github_utilg=get_github_util# query graphqlres=g.query_gql(graph_ql)# get file in github repositorybyte_object=g.download_from_git(github_url_path)Installation$pipinstallskt--upgradeIf you would like to install submodules for AIR$pipinstallskt[air]--upgradeDevelopCreate issue first and follow the GitHub flowhttps://help.github.com/en/github/collaborating-with-issues-and-pull-requests/github-flowAIPS EDA toolsOVERVIEWModeling EDA시 활용할 수 있는 기능의 공통 moduleModulesEDA (Nuemric / Categorical variable)1) EDA1. Numeric variable EDAdefnumeric_eda_plotNumeric feature에 대한 EDA Plot function

    Args. :
        - df           :   Pandas DataFrame 형태의 EDA대상 데이터
        - feature_list :   EDA 대상 feature list (df의 columns)
        - label_col    :   Label(or Hue) column
        - cols         :   Multi-plot 시 grid column 개수 (row 개수는 feature_list에 따라 자동으로 결정 됨)
        - n_samples    :   Label 별 sampling 할 개수 (default = -1(전수 데이터로 EDA할 경우))
        - plot_type    :   density or box (default = 'density')
        - stat_yn      :   기초 통계량 출력여부 (mean / min / max / 1q / 3q) (default : False)
        - figsize      :   (default : (7,4))

    Returns : 
        matplotlib.pyplot object

    Example : 
        fig = numeric_eda_plot(df, ['age'], 'answer', cols = 1, n_samples = 10000, plot_type='density', stat_yn=True, figsize = (7,4))
        fig

        if want to Save the EDA images,
        fig.savefig('filename')2. Categorical variable EDAdefcategorical_eda_plotCategorical feature에 대한 EDA Plot function

    Args. :
        - df           :   Pandas DataFrame 형태의 EDA대상 데이터
        - feature_list :   EDA 대상 feature list (df의 columns)
        - label_col    :   Label(or Hue) column
        - cols         :   Multi-plot 시 grid column 개수 (row 개수는 feature_list에 따라 자동으로 결정 됨)
        - n_samples    :   Label 별 sampling 할 개수 (default = -1(전수 데이터로 EDA할 경우))
        - figsize      :   (default : (7,4))

    Returns : 
        matplotlib.pyplot object


    Example : 
        Example : 
        fig = categorical_eda_plot(df, ['sex_cd'], 'answer', cols = 1, n_samples = 10000, figsize = (7,4))
        fig

        if want to Save the EDA images,
        fig.savefig('filename')