Package: joes-giant-toolbox
Joe's Giant Tool Boxhttps://github.com/J-sephB-lt-n/joes_giant_toolboxA large collection of general python functions and classes that I use in my daily work.-.
                                                    /   \
                                     _____.....-----|(o) |
                               _..--'          _..--|  .''
                             .'  o      _..--''     |  | |
                            /  _/_..--''            |  | |
                   ________/  / /                   |  | |
                  | _  ____\ / /                    |  | |
 _.-----._________|| ||    \\ /                     |  | |
|=================||=||_____\\                      |__|-'
|                 ||_||_____//                      (o\ |
|_________________|_________/                        |-\|
 `-------------._______.----'                        /  `.
    .,.,.,.,.,.,.,.,.,.,.,.,.,                      /     \
   ((O) o o o o ======= o o(O))                 ._.'      /
LGB `-.,.,.,.,.,.,.,.,.,.,.,-'                   `.......'source:https://ascii.co.uk(this badge lags by 1 release on pypi)Installationpipinstalljoes-giant-toolboxUsageThe scripts exist at varying levels of completeness (some have seen extensive use in many projects whereas others have been used little or have incomplete documentation and missing unit tests). In order to measure this, I have added in a confidence score for each:Confidence ScoreDescription5Code has been used (without any observed failures) in multiple production environments (or large real world projects)4Code has been used (without any observed failures) in a production environment (or large real world project)3Code appears to work perfectly and passes a suite of unit tests but has not yet been used in a production environment or large real world project2The code appears to work perfectly but has not been thoroughly tested1Skeleton of function/class is present but the code does not work fully yetYou can search by category:API and WebData VisualisationGoogle CloudProject ManagmentPython Convenience FunctionsStatistical Inference and Hypothesis TestingStatistical Modelling and Machine LearningText and Natural Language Processing..or you can just scroll through the master list:NameDescriptionConfidence Scoreanonymous_view_public_linkedin_pageExtracts the information (HTML) from a public LinkedIn page (e.g. person or company) using a virtual browser4ascii_density_histogramDraws a histogram using only raw text symbols2conjugate_prior_beta_binomialCalculates the posterior distribution of the success probability parameter [p] of a binomial distribution, from observed data and a user-specified beta prior4cosine_similarityCalculates the cosine similarity between two 1-dimensional numpy arrays2create_gcloud_vm_docker_templateCreates a folder containing the files necessary to quickly build a python docker container to run on a google cloud Virtual Machine4create_parallel_google_cloud_run_job_templateRun a task in parallel using a Google Cloud Run job (code-generating function)2create_project_scope_docCreates a basic project scope document (markdown) by prompting the user for input3DataBatcherBreaks a provided iterable up into batches according to a provided batching pattern4delete_file_in_gcloud_bucketDeletes a file which is in a google cloud bucket4download_file_from_gcloud_bucket_to_pythonReads a file from a google cloud bucket into python memory4duckduckgo_search_multipageFetches search results from the DuckDuckGo Lite search engine2gcloud_vm_deletes_itselfRunning this function on a google cloud Virtual Machine (VM) causes the VM to delete itself4list_all_python_importsSearches every python script in a given folder and lists all python modules imported within those scripts2list_files_in_gcloud_bucketReturns a list of the files present in a specified google cloud bucket4longest_common_substringIdentifies the longest substring appearing in both strings3longest_sentence_subsequence_plagiarism_detectorFinds phrases (sequences of consecutive words) common to 2 documents (e.g. to act as a naive plagiarism detector)3make_url_requestA convenience function for making API requests using the urllib library3move_or_rename_file_in_gcloud_bucketMove or rename a file which is in a google cloud bucket (which includes moving it to a different bucket)4parse_mime_email_partsExtracts parts from an email that is in MIME format2print_progress_barPrints a progress bar (to standard out) while code is running3PythonPlottingTutorialsExample code snippets for creating common data visualisations in python4query_bigquery_to_pandas_dfRuns a query on Google BigQuery and writes the result into a local pandas.DataFrame4RapidBinaryClassifierUltra rapid generation of binary classifier models in scikit-learn by abstracting away a lot of the decisions and model code3RegexRulesClassifierA multi-class text classifier using manual regex rules2require_api_keyA decorator adding basic API key authentication to a flask route3run_python_function_in_parallelRuns a python function in parallel on multiple cores or threads4scrape_webpage_and_all_linked_webpagesExtracts HTML from given web page, and also follows all of the hyperlinks on that page and scrapes those too1StringCleanerPerforms common string-cleaning operations to a text string, also allowing them to be chained in sequence1upload_file_python_to_gcloud_bucketWrites an object in python memory to a file (blob) on a google cloud bucket4url_to_filename_to_url_mapperConverts a webpage URL into a useable filename, where the URL can be recovered directly from the filename2view_nested_dict_structureGenerates a simple printout for understanding the structure of a complex nested python dictionary4write_pandas_df_to_google_bigquery_tableWrites a pandas dataframe to a table on Google BigQuery4API and Webimportjoes_giant_toolbox.webhelp(joes_giant_toolbox.web.anonymous_view_public_linkedin_page)help(joes_giant_toolbox.web.duckduckgo_search_multipage)help(joes_giant_toolbox.web.make_url_request)help(joes_giant_toolbox.web.require_api_key)help(joes_giant_toolbox.web.parse_mime_email_parts)help(joes_giant_toolbox.web.scrape_webpage_and_all_linked_webpages)help(joes_giant_toolbox.web.url_to_filename_to_url_mapper)NameDescriptionConfidence Scoreanonymous_view_public_linkedin_pageExtracts the information (HTML) from a public LinkedIn page (e.g. person or company) using a virtual browser2duckduckgo_search_multipageFetches search results from the DuckDuckGo Lite search engine2make_url_requestA convenience function for making API requests using the urllib library3parse_mime_email_partsExtracts parts from an email that is in MIME format2require_api_keyA decorator adding basic API key authentication to a flask route3scrape_webpage_and_all_linked_webpagesExtracts HTML from given web page, and also follows all of the hyperlinks on that page and scrapes those too1url_to_filename_to_url_mapperConverts a webpage URL into a useable filename, where the URL can be recovered directly from the filename2Data Visualisationimportjoes_giant_toolbox.datavizhelp(joes_giant_toolbox.dataviz)help(joes_giant_toolbox.dataviz.ascii_density_histogram)help(joes_giant_toolbox.dataviz.PythonPlottingTutorials)help(joes_giant_toolbox.dataviz.view_nested_dict_structure)NameDescriptionConfidence Scoreascii_density_histogramDraws a histogram using only raw text symbols2PythonPlottingTutorialsExample code snippets for creating common data visualisations in python4view_nested_dict_structureGenerates a simple printout for understanding the structure of a complex nested python dictionary4Google CloudTo additionally install the package dependencies of this module:pipinstalljoes-giant-toolbox[google]importjoes_giant_toolbox.google_cloudhelp(joes_giant_toolbox.google_cloud)help(joes_giant_toolbox.google_cloud.create_gcloud_vm_docker_template)help(joes_giant_toolbox.google_cloud.create_parallel_google_cloud_run_job_template)help(joes_giant_toolbox.google_cloud.delete_file_in_gcloud_bucket)help(joes_giant_toolbox.google_cloud.download_file_from_gcloud_bucket_to_python)help(joes_giant_toolbox.google_cloud.gcloud_vm_deletes_itself)help(joes_giant_toolbox.google_cloud.list_files_in_gcloud_bucket)help(joes_giant_toolbox.google_cloud.move_or_rename_file_in_gcloud_bucket)help(joes_giant_toolbox.google_cloud.query_bigquery_to_pandas_df)help(joes_giant_toolbox.google_cloud.upload_file_python_to_gcloud_bucket)help(joes_giant_toolbox.google_cloud.write_pandas_df_to_google_bigquery_table)NameDescriptionConfidence Scorecreate_gcloud_vm_docker_templateCreates a folder containing the files necessary to quickly build a python docker container to run on a google cloud Virtual Machine4create_parallel_google_cloud_run_job_templateRun a task in parallel using a Google Cloud Run job (code-generating function)2delete_file_in_gcloud_bucketDeletes a file which is in a google cloud bucket4download_file_from_gcloud_bucket_to_pythonReads a file from a google cloud bucket into python memory4gcloud_vm_deletes_itselfRunning this function on a google cloud Virtual Machine (VM) causes the VM to delete itself4list_files_in_gcloud_bucketReturns a list of the files present in a specified google cloud bucket4move_or_rename_file_in_gcloud_bucketMove or rename a file which is in a google cloud bucket (which includes moving it to a different bucket)4query_bigquery_to_pandas_dfRuns a query on Google BigQuery and writes the result into a local pandas.DataFrame4upload_file_python_to_gcloud_bucketWrites an object in python memory to a file (blob) on a google cloud bucket4write_pandas_df_to_google_bigquery_tableWrites a pandas dataframe to a table on Google BigQuery4Project Managementimportjoes_giant_toolbox.proj_mgmthelp(joes_giant_toolbox.proj_mgmt.create_project_scope_doc)NameDescriptionConfidence Scorecreate_project_scope_docCreates a basic project scope document (markdown) by prompting the user for input3Python Convenience Functionsimportjoes_giant_toolbox.conveniencehelp(joes_giant_toolbox.convenience.DataBatcher)help(joes_giant_toolbox.convenience.list_all_python_imports)help(joes_giant_toolbox.convenience.print_progress_bar)help(joes_giant_toolbox.convenience.run_python_function_in_parallel)NameDescriptionConfidence ScoreDataBatcherBreaks a provided iterable up into batches according to a provided batching pattern4list_all_python_importsSearches every python script in a given folder and lists all python modules imported within those scripts2print_progress_barPrints a progress bar (to standard out) while code is running3run_python_function_in_parallelRuns a python function in parallel on multiple cores or threads4Statistical Inference and Hypothesis Testingimportjoes_giant_toolbox.statshelp(joes_giant_toolbox.stats)help(joes_giant_toolbox.stats.conjugate_prior_beta_binomial)NameDescriptionConfidence Scoreconjugate_prior_beta_binomialCalculates the posterior distribution of the success probability parameter [p] of a binomial distribution, from observed data and a user-specified beta prior4Statistical Modelling and Machine Learningimportjoes_giant_toolbox.mathshelp(joes_giant_toolbox.maths.cosine_similarity)importjoes_giant_toolbox.sklearnhelp(joes_giant_toolbox.sklearn.RapidBinaryClassifier)NameDescriptionConfidence Scorecosine_similarityCalculates the cosine similarity between two 1-dimensional numpy arrays2RapidBinaryClassifierUltra rapid generation of binary classifier models in scikit-learn by abstracting away a lot of the decisions and model code3Text and Natural Language Processingimportjoes_giant_toolbox.texthelp(joes_giant_toolbox.text)help(joes_giant_toolbox.text.longest_common_substring)help(joes_giant_toolbox.text.longest_sentence_subsequence_plagiarism_detector)help(joes_giant_toolbox.text.RegexRulesClassifier)help(joes_giant_toolbox.text.StringCleaner)NameDescriptionConfidence Scorelongest_common_substringIdentifies the longest substring appearing in both strings3longest_sentence_subsequence_plagiarism_detectorFinds phrases (sequences of consecutive words) common to 2 documents (e.g. to act as a naive plagiarism detector)3RegexRulesClassifierA multi-class text classifier using manual regex rules2StringCleanerPerforms common string-cleaning operations to a text string, also allowing them to be chained in sequence1Run Unit Testspipinstallpytestcdjoes_giant_toolbox/tests
pytest--verbose