Package: fastHDMI
fastHDMI -- fast High-Dimensional Mutual Information estimationKai Yang<kai.yang2 "at" mail.mcgill.ca>License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)GPG Public key Fingerprint: CC02CF153594774CF956691492B2600D18170329This packages uses FFT-based mutual information screening and accelerated gradient method for important variables from (potentially very) high-dimensional large datasets.Consider the sizes of the datafiles, the most commonly-used functions are the functions run in parallel -- all functions running in parallel will has_parallelsuffix; and they all have arguments:core_num: number of CPU cores used for multiprocessing; the default option is to use all the cores available, considering the job is most likely running on a server instead of a PCmultp: job multiplier, the job to be run in parallel will be first divided intocore_num * multpsub-jobs -- as equal as possible, then at each time, one core will take one subjob.verbose: how verbal the function will be, with0being least verbal and increases wrt. the number decalred hereThe function implementing our propsoed FFT-based mutual information estimation will have the following arguments:N: the grid size for 1-D FFT; withN=500as the default valuea_N,a_N: similar to above, the grid size for 2-D FFT; with300as the default valueskernelandbwspecify the kernel and bandwidth used for KDEnormis the norm used for KDE -- this option only takes effects for 2-D KDEThe screening functions and their arguments:Forplinkfiles:argumentsbed_file,bim_file,fam_fileare the location of the plink files;argumentsoutcome,outcome_iidare the outcome values and the iids for the outcome. For genetic data, it is usual that the order of SNP iid and the outcome iid don't match. While SNP iid can be obtained from the plink1 files, outcome iid here is to be declared separately.outcome_iidshould be a list of strings or a one-dimensional numpy string array.continuous_screening_plink,continuous_screening_plink_parallelfor screening on continuous outcomes with continuous covariatesbinary_screening_plink,binary_screening_plink_parallelfor screening on binary outcomes with continuous covariatesclump_plink_parallelfor clumping -- starting from the first covariate (i.e., the first column on the left of the datafile), clumping will remove all subsequent covariates with a mutual information higher than what theclumping_thresholddeclares with the one it looks atForcsvfiles:argument_usecolsis a list of column labels to be used,the first element should be the outcome. Returned mutual information calculation results match_usecols.Note that it is assumed the left first column should be the outcome;if not, use_usecolsto set the first element to be the outcome column label.csv_enginecan usedaskfor low memory situations, orpandas'sread_csvengines, orfastparquetengine for a createdparquetfile for faster speed. Iffastparquetis chosen, declareparquet_fileas the filepath to the parquet file; ifdaskis chosen to read very large CSV, it might need to specify a largersample.binary_screening_csv,binary_screening_csv_parallelfor screening on binary outcomes with continuous covariatesbinary_skMI_screening_csv_parallel,continuous_skMI_screening_csv_parallelfor screening using mutual information estimation provided byskLearn, i.e.,sklearn.metrics.mutual_info_score,sklearn.feature_selection.mutual_info_classifPearson_screening_csv_parallelfor screening using Pearson correlationcontinuous_screening_csv,continuous_screening_csv_parallelfor screening on continuous outcomes with continuous covariatesclump_continuous_csv_parallelsimilar to aboveAshare_memoryoption is added for multiprocess computing. As a feature, it can be applied on large.csvdata in parallel in a memory-efficient manner and use FFT for KDE to estimate the mutual information extremely fast. A tqdm progress bar is now added to be more useful on cloud computing platforms.verboseoption can take values of0,1,2, with2being most verbal;1being only show progress bar, and0being not verbal at all.For DataFrame files:binary_screening_dataframe,binary_screening_dataframe_parallelfor screening on binary outcomes with continuous covariatesbinary_skMI_screening_dataframe_parallel,continuous_skMI_screening_dataframe_parallelfor screening using mutual information estimation provided byskLearn, i.e.,sklearn.metrics.mutual_info_score,sklearn.feature_selection.mutual_info_classifPearson_screening_dataframe_parallelfor screening using Pearson correlationcontinuous_screening_dataframe,continuous_screening_dataframe_parallelfor screening on continuous outcomes with continuous covariatesclump_continuous_dataframe_parallelsimilar to aboveFornumpyarrays:binary_screening_array,binary_screening_array_parallelfor screening on binary outcomes with continuous covariatescontinuous_screening_array,continuous_screening_array_parallelfor screening on continuous outcomes with continuous covariatesbinary_skMI_array_parallel,continuous_skMI_array_parallelfor screening using mutual information estimation provided byskLearn, i.e.,sklearn.metrics.mutual_info_score,sklearn.feature_selection.mutual_info_classifcontinuous_Pearson_array_parallelfor screening using Pearson correlation