Package: analysistoolbox
Analysis Tool BoxDescriptionAnalysis Tool Box (i.e. "analysistoolbox") is a collection of tools in Python for data collection and processing, statisitics, analytics, and intelligence analysis.Getting StartedTo install the package, run the following command in the root directory of the project:pipinstallanalysistoolboxVisualizations are created using the matplotlib and seaborn libraries. While you can select whichever seaborn style you'd like, the following Seaborn style tends to get the best looking plots:sns.set(style="white",font="Arial",context="paper")UsageThere are many modules in the analysistoolbox package, each with their own functions. The following is a list of the modules:CalculusData collectionData processingDescriptive analyticsFile managementHypothesis testingLinear algebraPredictive analyticsStatisticsVisualizationsCalculusThere are several functions in the Calculus submodule. The following is a list of the functions:FindDerivativeFindLimitOfFunctionFindMinimumSquareLossPlotFunctionFindDerivativeTheFindDerivativefunction calculates the derivative of a given function. It uses the sympy library, a Python library for symbolic mathematics, to perform the differentiation. The function also has the capability to print the original function and its derivative, return the derivative function, and plot both the original function and its derivative.# Load the FindDerivative function from the Calculus submodulefromanalysistoolbox.calculusimportFindDerivativeimportsympy# Define a symbolic variablex=sympy.symbols('x')# Define a functionf_of_x=x**3+2*x**2+3*x+4# Use the FindDerivative functionFindDerivative(f_of_x,print_functions=True,return_derivative_function=True,plot_functions=True)FindLimitOfFunctionTheFindLimitOfFunctionfunction finds the limit of a function at a specific point and optionally plot the function and its tangent line at that point. The script uses the matplotlib and numpy libraries for plotting and numerical operations respectively.# Import the necessary librariesfromanalysistoolbox.calculusimportFindLimitOfFunctionimportnumpyasnpimportsympy# Define a symbolic variablex=sympy.symbols('x')# Define a functionf_of_x=np.sin(x)/x# Use the FindLimitOfFunction functionFindLimitOfFunction(f_of_x,point=0,step=0.01,plot_function=True,x_minimum=-10,x_maximum=10,n=1000,tangent_line_window=1)FindMinimumSquareLossTheFindMinimumSquareLossfunction calculates the minimum square loss between observed and predicted values. This function is often used in machine learning and statistics to measure the average squared difference between the actual and predicted outcomes.# Import the necessary librariesfromanalysistoolbox.calculusimportFindMinimumSquareLoss# Define observed and predicted valuesobserved_values=[1,2,3,4,5]predicted_values=[1.1,1.9,3.2,3.7,5.1]# Use the FindMinimumSquareLoss functionminimum_square_loss=FindMinimumSquareLoss(observed_values,predicted_values,show_plot=True)# Print the minimum square lossprint(f"The minimum square loss is:{minimum_square_loss}")PlotFunctionThePlotFunctionfunction plots a mathematical function of x. It takes a lambda function as input and allows for customization of the plot.# Import the necessary librariesfromanalysistoolbox.calculusimportPlotFunctionimportsympy# Set x as a symbolic variablex=sympy.symbols('x')# Define the function to plotf_of_x=lambdax:x**2# Plot the function with default settingsPlotFunction(f_of_x)Data CollectionThere are several functions in the Data Collection submodule. The following is a list of the functions:ExtractTextFromPDFFetchPDFFromURLFetchUSShapefileFetchWebsiteTextGetGoogleSearchResultsGetZipFileExtractTextFromPDFTheExtractTextFromPDFfunction extracts text from a PDF file, cleans it, then saves it to a text file.# Import the functionfromanalysistoolbox.data_collectionimportExtractTextFromPDF# Call the functionExtractTextFromPDF(filepath_to_pdf="/path/to/your/input.pdf",filepath_for_exported_text="/path/to/your/output.txt",start_page=1,end_page=None)FetchPDFFromURLTheFetchPDFFromURLfunction downloads a PDF file from a URL and saves it to a specified location.# Import the functionfromanalysistoolbox.data_collectionimportFetchPDFFromURL# Call the function to download the PDFFetchPDFFromURL(url="https://example.com/sample.pdf",filename="C:/folder/sample.pdf")FetchUSShapefileTheFetchUSShapefilefunction fetches a geographical shapefile from the TIGER database of the U.S. Census Bureau.# Import the functionfromanalysistoolbox.data_collectionimportFetchUSShapefile# Fetch the shapefile for the census tracts in King County, Washington, for the 2021 census yearshapefile=FetchUSShapefile(state='PA',county='Allegheny',geography='tract',census_year=2021)# Print the first few rows of the shapefileprint(shapefile.head())FetchWebsiteTextTheFetchWebsiteTextfunction fetches the text from a website and saves it to a text file.# Import the functionfromanalysistoolbox.data_collectionimportFetchWebsiteText# Call the functiontext=FetchWebsiteText(url="https://www.example.com",browserless_api_key="your_browserless_api_key")# Print the fetched textprint(text)GetGoogleSearchResultsTheGetGoogleSearchResultsfunction fetches Google search results for a given query using the Serper API.# Import the functionfromanalysistoolbox.data_collectionimportGetGoogleSearchResults# Call the function with the query# Make sure to replace 'your_serper_api_key' with your actual Serper API keyresults=GetGoogleSearchResults(query="Python programming",serper_api_key='your_serper_api_key',number_of_results=5,apply_autocorrect=True,display_results=True)# Print the resultsprint(results)GetZipFileTheGetZipFilefunction downloads a zip file from a url and saves it to a specified folder. It can also unzip the file and print the contents of the zip file.# Import the functionfromanalysistoolbox.data_collectionimportGetZipFile# Call the functionGetZipFile(url="http://example.com/file.zip",path_to_save_folder="/path/to/save/folder")Data ProcessingThere are several functions in the Data Processing submodule. The following is a list of the functions:AddDateNumberColumnsAddLeadingZerosAddRowCountColumnAddTPeriodColumnAddTukeyOutlierColumnCleanTextColumnsConductAnomalyDetectionConductEntityMatchingConvertOddsToProbabilityCountMissingDataByGroupCreateBinnedColumnCreateDataOverviewCreateRandomSampleGroupsCreateRareCategoryColumnCreateStratifiedRandomSampleGroupsImputeMissingValuesUsingNearestNeighborsVerifyGranularityAddDateNumberColumnsTheAddDateNumberColumnsfunction adds columns for the year, month, quarter, week, day of the month, and day of the week to a dataframe.# Import necessary packagesfromanalysistoolbox.data_processingimportAddDateNumberColumnsfromdatetimeimportdatetimeimportpandasaspd# Create a sample dataframedata={'Date':[datetime(2020,1,1),datetime(2020,2,1),datetime(2020,3,1),datetime(2020,4,1)]}df=pd.DataFrame(data)# Use the function on the sample dataframedf=AddDateNumberColumns(dataframe=df,date_column_name='Date')# Print the updated dataframeprint(df)AddLeadingZerosTheAddLeadingZerosfunction adds leading zeros to a column. If fixed_length is not specified, the longest string in the column is used as the fixed length. If add_as_new_column is set to True, the new column is added to the dataframe. Otherwise, the original column is updated.# Import necessary packagesfromanalysistoolbox.data_processingimportAddLeadingZerosimportpandasaspd# Create a sample dataframedata={'ID':[1,23,456,7890]}df=pd.DataFrame(data)# Use the AddLeadingZeros functiondf=AddLeadingZeros(dataframe=df,column_name='ID',add_as_new_column=True)# Print updated dataframeprint(df)AddRowCountColumnTheAddRowCountColumnfunction adds a column to a dataframe that contains the row number for each row, based on a group (or groups) of columns. The function can also sort the dataframe by a column or columns before adding the row count column.# Import necessary packagesfromanalysistoolbox.data_processingimportAddRowCountColumnimportpandasaspd# Create a sample dataframedata={'Payment Method':['Check','Credit Card','Check','Credit Card','Check','Credit Card','Check','Credit Card'],'Transaction Value':[100,200,300,400,500,600,700,800],'Transaction Order':[1,2,3,4,5,6,7,8]}df=pd.DataFrame(data)# Call the functiondf_updated=AddRowCountColumn(dataframe=df,list_of_grouping_variables=['Payment Method'],list_of_order_columns=['Transaction Order'],list_of_ascending_order_args=[True])# Print the updated dataframeprint(df_updated)AddTPeriodColumnTheAddTPeriodColumnfunction adds a T-period column to a dataframe. The T-period column is the number of intervals (e.g., days or weeks) since the earliest date in the dataframe.# Import necessary librariesfromanalysistoolbox.data_processingimportAddTPeriodColumnfromdatetimeimportdatetimeimportpandasaspd# Create a sample dataframedata={'date':pd.date_range(start='1/1/2020',end='1/10/2020'),'value':range(1,11)}df=pd.DataFrame(data)# Use the functiondf_updated=AddTPeriodColumn(dataframe=df,date_column_name='date',t_period_interval='days')# Print the updated dataframeprint(df_updated)AddTukeyOutlierColumnTheAddTukeyOutlierColumnfunction adds a column to a dataframe that indicates whether a value is an outlier. The function uses the Tukey method to identify outliers.# Import necessary librariesfromanalysistoolbox.data_processingimportAddTukeyOutlierColumnimportpandasaspd# Create a sample dataframedata=pd.DataFrame({'values':[1,2,3,4,5,6,7,8,9,20]})# Use the functiondf_updated=AddTukeyOutlierColumn(dataframe=data,value_column_name='values',tukey_boundary_multiplier=1.5,plot_tukey_outliers=True)# Print the updated dataframeprint(df_updated)CleanTextColumnsTheCleanTextColumnsfunction cleans string-type columns in a pandas DataFrame by removing all leading and trailing spaces.# Import necessary librariesfromanalysistoolbox.data_processingimportCleanTextColumnsimportpandasaspd# Create a sample dataframedf=pd.DataFrame({'A':[' hello','world ',' python '],'B':[1,2,3],})# Clean the dataframedf_clean=CleanTextColumns(df)ConductAnomalyDetectionTheConductAnomalyDetectionfunction performs anomaly detection on a given dataset using the z-score method.# Import necessary librariesfromanalysistoolbox.data_processingimportConductAnomalyDetectionimportpandasaspd# Create a sample dataframedf=pd.DataFrame({'A':[1,2,3,1000],'B':[4,5,6,2000],})# Conduct anomaly detectiondf_anomaly_detected=ConductAnomalyDetection(dataframe=df,list_of_columns_to_analyze=['A','B'])# Print the updated dataframeprint(df_anomaly_detected)ConductEntityMatchingTheConductEntityMatchingfunction performs entity matching between two dataframes using various fuzzy matching algorithms.fromanalysistoolbox.data_processingimportConductEntityMatchingimportpandasaspd# Create two dataframesdataframe_1=pd.DataFrame({'ID':['1','2','3'],'Name':['John Doe','Jane Smith','Bob Johnson'],'City':['New York','Los Angeles','Chicago']})dataframe_2=pd.DataFrame({'ID':['a','b','c'],'Name':['Jon Doe','Jane Smyth','Robert Johnson'],'City':['NYC','LA','Chicago']})# Conduct entity matchingmatched_entities=ConductEntityMatching(dataframe_1=dataframe_1,dataframe_1_primary_key='ID',dataframe_2=dataframe_2,dataframe_2_primary_key='ID',levenshtein_distance_filter=3,match_score_threshold=80,columns_to_compare=['Name','City'],match_methods=['Partial Token Set Ratio','Weighted Ratio'])ConvertOddsToProbabilityTheConvertOddsToProbabilityfunction converts odds to probability in a new column.# Import necessary packagesfromanalysistoolbox.data_processingimportConvertOddsToProbabilityimportpandasaspd# Create a sample dataframedata={'Team':['Team1','Team2','Team3','Team4'],'Odds':[2.5,1.5,3.0,np.nan]}df=pd.DataFrame(data)# Print the original dataframeprint("Original DataFrame:")print(df)# Use the function to convert odds to probabilitydf=ConvertOddsToProbability(dataframe=df,odds_column='Odds')CountMissingDataByGroupTheCountMissingDataByGroupfunction counts the number of records with missing data in a Pandas dataframe, grouped by specified columns.# Import necessary packagesfromanalysistoolbox.data_processingimportCountMissingDataByGroupimportpandasaspdimportnumpyasnp# Create a sample dataframe with some missing valuesdata={'Group':['A','B','A','B','A','B'],'Value1':[1,2,np.nan,4,5,np.nan],'Value2':[np.nan,8,9,10,np.nan,12]}df=pd.DataFrame(data)# Use the function to count missing data by groupCountMissingDataByGroup(dataframe=df,list_of_grouping_columns=['Group'])CreateBinnedColumnTheCreateBinnedColumnfunction creates a new column in a Pandas dataframe based on a numeric variable. Binning is a process of transforming continuous numerical variables into discrete categorical 'bins'.# Import necessary packagesfromanalysistoolbox.data_processingimportCreateBinnedColumnimportpandasaspdimportnumpyasnp# Create a sample dataframedata={'Group':['A','B','A','B','A','B'],'Value1':[1,2,3,4,5,6],'Value2':[7,8,9,10,11,12]}df=pd.DataFrame(data)# Use the function to create a binned columndf_binned=CreateBinnedColumn(dataframe=df,numeric_column_name='Value1',number_of_bins=3,binning_strategy='uniform')CreateDataOverviewTheCreateDataOverviewfunction creates an overview of a Pandas dataframe, including the data type, missing count, missing percentage, and summary statistics for each variable in the DataFrame.# Import necessary packagesfromanalysistoolbox.data_processingimportCreateDataOverviewimportpandasaspdimportnumpyasnp# Create a sample dataframedata={'Column1':[1,2,3,np.nan,5,6],'Column2':['a','b','c','d',np.nan,'f'],'Column3':[7.1,8.2,9.3,10.4,np.nan,12.5]}df=pd.DataFrame(data)# Use the function to create an overview of the dataframeCreateDataOverview(dataframe=df,plot_missingness=True)CreateRandomSampleGroupsTheCreateRandomSampleGroupsfunction a takes a pandas DataFrame, shuffle its rows, assign each row to one of n groups, and then return the updated DataFrame with an additional column indicating the group number.# Import necessary packagesfromanalysistoolbox.data_processingimportCreateRandomSampleGroupsimportpandasaspd# Create a sample DataFramedata={'Name':['Alice','Bob','Charlie','David','Eve'],'Age':[25,31,35,19,45],'Score':[85,95,78,81,92]}df=pd.DataFrame(data)# Use the functiongrouped_df=CreateRandomSampleGroups(dataframe=df,number_of_groups=2,random_seed=123)CreateRareCategoryColumnTheCreateRareCategoryColumnfunction creates a new column in a Pandas dataframe that indicates whether a categorical variable value is rare. A rare category is a category that occurs less than a specified percentage of the time.# Import necessary packagesfromanalysistoolbox.data_processingimportCreateRareCategoryColumnimportpandasaspd# Create a sample DataFramedata={'Name':['Alice','Bob','Charlie','David','Eve','Alice','Bob','Alice'],'Age':[25,31,35,19,45,23,30,24],'Score':[85,95,78,81,92,88,90,86]}df=pd.DataFrame(data)# Use the functionupdated_df=CreateRareCategoryColumn(dataframe=df,categorical_column_name='Name',rare_category_label='Rare',rare_category_threshold=0.05,new_column_suffix='(rare category)')CreateStratifiedRandomSampleGroupsTheCreateStratifiedRandomSampleGroupsunction performs stratified random sampling on a pandas DataFrame. Stratified random sampling is a method of sampling that involves the division of a population into smaller groups known as strata. In stratified random sampling, the strata are formed based on members' shared attributes or characteristics.# Import necessary packagesfromanalysistoolbox.data_processingimportCreateStratifiedRandomSampleGroupsimportnumpyasnpimportpandasaspd# Create a sample DataFramedata={'Name':['Alice','Bob','Charlie','David','Eve','Alice','Bob','Alice'],'Age':[25,31,35,19,45,23,30,24],'Score':[85,95,78,81,92,88,90,86]}df=pd.DataFrame(data)# Use the functionstratified_df=CreateStratifiedRandomSampleGroups(dataframe=df,number_of_groups=2,list_categorical_column_names=['Name'],random_seed=42)ImputeMissingValuesUsingNearestNeighborsTheImputeMissingValuesUsingNearestNeighborsfunction imputes missing values in a dataframe using the nearest neighbors method. For each sample with missing values, it finds the n_neighbors nearest neighbors in the training set and imputes the missing values using the mean value of these neighbors.# Import necessary packagesfromanalysistoolbox.data_processingimportImputeMissingValuesUsingNearestNeighborsimportpandasaspdimportnumpyasnp# Create a sample DataFrame with missing valuesdata={'A':[1,2,np.nan,4,5],'B':[np.nan,2,3,4,5],'C':[1,2,3,np.nan,5],'D':[1,2,3,4,np.nan]}df=pd.DataFrame(data)# Use the functionimputed_df=ImputeMissingValuesUsingNearestNeighbors(dataframe=df,list_of_numeric_columns_to_impute=['A','B','C','D'],number_of_neighbors=2,averaging_method='uniform')VerifyGranularityTheVerifyGranularityfunction checks the granularity of a given dataframe based on a list of key columns. Granularity in this context refers to the level of detail or summarization in a set of data.# Import necessary packagesfromanalysistoolbox.data_processingimportVerifyGranularityimportpandasaspd# Create a sample DataFramedata={'Name':['Alice','Bob','Charlie','David','Eve','Alice','Bob','Alice'],'Age':[25,31,35,19,45,23,30,24],'Score':[85,95,78,81,92,88,90,86]}df=pd.DataFrame(data)# Use the functionVerifyGranularity(dataframe=df,list_of_key_columns=['Name','Age'],set_key_as_index=True,print_as_markdown=False)Descriptive AnalyticsThere are several functions in the Descriptive Analytics submodule. The following is a list of the functions:ConductManifoldLearningConductPrincipalComponentAnalysisCreateAssociationRulesCreateGaussianMixtureClustersCreateHierarchicalClustersCreateKMeansClustersGenerateEDAWithLIDAConductManifoldLearningTheConductManifoldLearningfunction performs manifold learning on a given dataframe and returns a new dataframe with the original columns and the new manifold learning components. Manifold learning is a type of unsupervised learning that is used to reduce the dimensionality of the data.# Import necessary packagesfromanalysistoolbox.descriptive_analyticsimportConductManifoldLearningimportpandasaspdfromsklearn.datasetsimportload_iris# Load the iris datasetiris=load_iris()iris_df=pd.DataFrame(data=iris.data,columns=iris.feature_names)# Use the functionnew_df=ConductManifoldLearning(dataframe=iris_df,list_of_numeric_columns=['sepal length (cm)','sepal width (cm)','petal length (cm)','petal width (cm)'],number_of_components=2,random_seed=42,show_component_summary_plots=True,summary_plot_size=(10,10))ConductPrincipalComponentAnalysisTheConductPrincipalComponentAnalysisfunction performs Principal Component Analysis (PCA) on a given dataframe. PCA is a technique used in machine learning to reduce the dimensionality of data while retaining as much information as possible.# Import necessary packagesfromanalysistoolbox.descriptive_analyticsimportConductManifoldLearningimportpandasaspdfromsklearn.datasetsimportload_iris# Load the iris datasetiris=load_iris()iris_df=pd.DataFrame(data=iris.data,columns=iris.feature_names)# Call the functionresult=ConductPrincipalComponentAnalysis(dataframe=iris_df,list_of_numeric_columns=['sepal length (cm)','sepal width (cm)','petal length (cm)','petal width (cm)'],number_of_components=2)CreateAssociationRulesTheCreateAssociationRulesfunction creates association rules from a given dataframe. Association rules are widely used in market basket analysis, where the goal is to find associations and/or correlations among a set of items.# Import necessary packagesfromanalysistoolbox.descriptive_analyticsimportCreateAssociationRulesimportpandasaspd# Assuming you have a dataframe 'df' with 'TransactionID' and 'Item' columnsresult=CreateAssociationRules(dataframe=df,transaction_id_column='TransactionID',items_column='Item',support_threshold=0.01,confidence_threshold=0.2,plot_lift=True,plot_title='Association Rules',plot_size=(10,7))CreateGaussianMixtureClustersTheCreateGaussianMixtureClustersfunction creates Gaussian mixture clusters from a given dataframe. Gaussian mixture models are a type of unsupervised learning that is used to find clusters in data. It adds the resulting clusters as a new column in the dataframe, and also calculates the probability of each data point belonging to each cluster.# Import necessary packagesfromanalysistoolbox.descriptive_analyticsimportCreateGaussianMixtureClustersimportpandasaspdfromsklearnimportdatasets# Load the iris datasetiris=datasets.load_iris()# Convert the iris dataset to a pandas dataframedf=pd.DataFrame(data=np.c_[iris['data'],iris['target']],columns=iris['feature_names']+['target'])# Call the CreateGaussianMixtureClusters functiondf_clustered=CreateGaussianMixtureClusters(dataframe=df,list_of_numeric_columns_for_clustering=['sepal length (cm)','sepal width (cm)','petal length (cm)','petal width (cm)'],number_of_clusters=3,column_name_for_clusters='Gaussian Mixture Cluster',scale_predictor_variables=True,show_cluster_summary_plots=True,sns_color_palette='Set2',summary_plot_size=(15,15),random_seed=123,maximum_iterations=200)CreateHierarchicalClustersTheCreateHierarchicalClustersfunction creates hierarchical clusters from a given dataframe. Hierarchical clustering is a type of unsupervised learning that is used to find clusters in data. It adds the resulting clusters as a new column in the dataframe.# Import necessary packagesfromanalysistoolbox.descriptive_analyticsimportCreateHierarchicalClustersimportpandasaspdfromsklearnimportdatasets# Load the iris datasetiris=datasets.load_iris()df=pd.DataFrame(data=iris.data,columns=iris.feature_names)# Call the CreateHierarchicalClusters functiondf_clustered=CreateHierarchicalClusters(dataframe=df,list_of_value_columns_for_clustering=['sepal length (cm)','sepal width (cm)','petal length (cm)','petal width (cm)'],number_of_clusters=3,column_name_for_clusters='Hierarchical Cluster',scale_predictor_variables=True,show_cluster_summary_plots=True,color_palette='Set2',summary_plot_size=(6,4),random_seed=412,maximum_iterations=300)CreateKMeansClustersTheCreateKMeansClustersfunction performs K-Means clustering on a given dataset and returns the dataset with an additional column indicating the cluster each record belongs to.# Import necessary packagesfromanalysistoolbox.descriptive_analyticsimportCreateKMeansClustersimportpandasaspdfromsklearnimportdatasets# Load the iris datasetiris=datasets.load_iris()df=pd.DataFrame(data=iris.data,columns=iris.feature_names)# Call the CreateKMeansClusters functiondf_clustered=CreateKMeansClusters(dataframe=df,list_of_value_columns_for_clustering=['sepal length (cm)','sepal width (cm)','petal length (cm)','petal width (cm)'],number_of_clusters=3,column_name_for_clusters='K-Means Cluster',scale_predictor_variables=True,show_cluster_summary_plots=True,color_palette='Set2',summary_plot_size=(6,4),random_seed=412,maximum_iterations=300)GenerateEDAWithLIDATheGenerateEDAWithLIDAfunction uses the LIDA package from Microsoft to generate exploratory data analysis (EDA) goals.# Import necessary packagesfromanalysistoolbox.descriptive_analyticsimportGenerateEDAWithLIDAimportpandasaspdfromsklearnimportdatasets# Load the iris datasetiris=datasets.load_iris()df=pd.DataFrame(data=iris.data,columns=iris.feature_names)# Call the GenerateEDAWithLIDA functiondf_summary=GenerateEDAWithLIDA(dataframe=df,llm_api_key="your_llm_api_key_here",llm_provider="openai",llm_model="gpt-3.5-turbo",visualization_library="seaborn",goal_temperature=0.50,code_generation_temperature=0.05,data_summary_method="llm",number_of_samples_to_show_in_summary=5,return_data_fields_summary=True,number_of_goals_to_generate=5,plot_recommended_visualization=True,show_code_for_recommended_visualization=True)File ManagementHypothesis TestingLinear AlgebraPredictive AnalyticsPrescriptive AnalyticsSimulationsVisualizationsContributionsTo report an issue, request a feature, or contribute to the project, please see theCONTRIBUTING.mdfile (in progress).LicenseThis project is licensed under the MIT License - see theLICENSE.mdfile for details.