Package: thresher-py
Thresher - THRESHold EvaluatoR for PythonA bare pandas implementation of a tool for finding the threshold which maximizes accuracy
ofpredict_probalike-outputs (from e.g.scikit-learn), in regard to the provided ground truth (labels).Note: you can jump directly to the sample usagehere.Project descriptionMethod interesting for the user isoptimize_threshold(scores, actual_classes), which is available
from theThresherclass. This method, for givenscoresandactual classes,
returns a threshold that yields thehighest fractionof correctly classifiedsamples.optimize_threshold parameters:
  scores​:list
    The list of scores.
  actual_classes​:list
    The list of ground truth (correct) classes. 
    Classes are represented as -1 and 1.
returns:
  threshold:​float
    The threshold value that yields ​the highest fraction of correctly classified 
    samples​. If multiple thresholds give the optimal fraction, return any threshold.An oracle mechanismWe implemented a meta-optimizer - an 'oracle' mechanism, which chooses a proper algorithm in regard to the provided data. This is the default behaviour, and can be controlled by changing thealgorithmparam of theThresherconstructor. See the source code oforacle.pyandinterface.pyfor more details.Implemented algorithmsLinear searchThis is the most basic, iterative approach. Recommended for smaller datasets. For everythresholdpresent in the input (in thescoreslist), we evaluate it by calculating the exact accuracy ofsplitproduced by such threshold. Then, return the threshold which produce the most accurate split.List of parameters to customize:n_jobs(default: 1) - set to-1for using all available processors except one; any value of2or more
enables multiprocessing, while the default value of1disables multiprocessing2-dim Stochastic Gradient DescenttbdList of parameters to customize:num_of_iters(default: 200) - number of iterations during which algorithm tries to convergestop_thresh(default: 0.001) - minimal value of improvement, below which algorithm stopsalpha(default: 0.01)Evolutionary algorithmThis is a simulation approach which uses an evolutionary algorithm. It works by simulating multiple generations of a "population" of candidate solutions. During every iteration of a single generation, algorithm stochasticly evaluates the candidate solution. After the end of a single generation, we remove the from the population least fit agents (solutions), and do thecrossoverbetween the left solitions to produce new "offspring" candidate solutions. Moreover, they may mutate to provide additional random chance.List of parameters to customize:population_size(default: 30) - number of agents in the simulationnumber_of_generations(default: 20) - number of generationsnumber_of_iterations(default: 10) - number of iterations per a generationsus_factor(default: 2) - how many least-fit agents should be childless at the end of generationstoch_ratio(default: 0.02) - percentage of data to evaluate fit of a single agent per iterationoptimized_start(default: True)mutation_chance(default: 0.05)mutation_factor(default: 0.10)Grid searchAdded in version0.1.2. This algorithm works by generate a grid of possible solutions, with a granularity set
by parameter namedno_of_decimal_places. All candidate solutions are evaluated thoroughly
and the best one is chosen at the end.List of parameters to customize:no_of_decimal_places(default: 2) - generate the grid by rounding the number to the given number of decimal placesStochastic Grid searchAdded in version0.1.2. This algorithm works similarly like the above-mentioned 'Grid search' method, with the difference, that
every single point generated by the grid is evaluated only partially (which can be controlled by thestoch_ratioparameter)List of parameters to customize:no_of_decimal_places(default: 2) - generate the grid by rounding the number to the given number of decimal placesstoch_ratio(default: 0.05) - percentage of data to evaluate fit of a candidate number in the gridreshuffle(default: False) - set whether the random projection should be calculated every step, or notHow to setup?The process is rather straightforward, you just need to just whether to install
from the sources (latest revision), or from the PyPI repository (stable release).RequirementsTested with Python3.7+, on a standard Unix environmentInstallationInstallation from source:pip install git+https://github.com/oskar-j/thresher.gitStable release using thepiptool:pip install thresher-pyCustom parametersIt's possible to provide additional parameters in theThresherconstructor.Thresher(algorithm='auto',allow_parallel=True,verbose=False,progress_bar=False,labels=(0,1))Here is a description of what does every particular parameter do:algorithm(default value:'auto') - allows to manually choose the algorithm from the list of available algorithms.
Same effect can be achieved with running the method calledset_algorithm(algorithm_name)on theThresherinstance.
The default value is 'auto', which means that the tool uses an oracle mechanism to manually choose a proper algorithm.allow_parallel(default value:True) - enables/disabled multiprocessing for algorithmsverbose(default value:False) - enables verbosityprogress_bar(default value:False) - shows a progress bar in the terminal (if supported by the algorithm)labels- necessary if your labels are different from(-1, 1)- first item from the tuple/list is a negative label,
and the second item is a positive labelControl parameters for the algorithmsSome of the above-mentioned algorithms allow to change their parameters.
They should be provided in a dictionary, inside thealgorithm_paramsparameter.
If no such customs parameters are provided, default values apply.Examples:t=thresher.Thresher(algorithm_params={'n_jobs':3})t=thresher.Thresher(algorithm_params={'no_of_decimal_places':3,'stoch_ratio':0.10})Sample usageimportthreshert=thresher.Thresher()print('Currently supported algorithms:')print(t.get_supported_algorithms())cases=[0.1,0.3,0.4,0.7]actual_labels=[-1,-1,1,1]print(f'Optimization result:{t.optimize_threshold(cases,actual_labels)}')See theexamplesdirectory for more sample code.Performance testsA very basic performance test (with 10 repeats, on a real-worldanonymized dataconsisting of10^6rows) can be found in the Notebooklocated here.Future workadding more algorithms,publishing on conda,more heavy test loads,python docs,CI/CD pipeline for automated tests.