Package: task-substitution
task_substitutionSolve an auxiliary task using ML.This library is created by usingnbdev, please check it out.Task Substitutionis a method of solving an auxiliary problem ( with different features and different target ) in order to better understand the initial problem and solving it efficiently.Let's take a look at standard machine learning task, in the figure below you see a regression task with featuresf1,f2,f3and target variabley.We want to build on a model on the above dataset to predict for unknownyvalues in the test dataset shown below.Exploratory Data AnalysisFirst step we take to solve the problem is to look at the data, there can be many features withmissing valuesoroutlierswhich needs to be understood. It is possible that there is a relationship between a missing value and values of other features.Recover Missing ValuesIt is possible for a feature to have a missing value, it could be a data recording issue or bug etc. Often times for numerical features we replace missing value withmeanormedianvalue as a approximation. Sometimes we replace missing value with values like-9999so that model treats them differently or sometimes we leave them as is as libraries likexgboostandlightgbmcan handlenulls. Let's look at following datasetHere we have a featuref3with missing values, this is a numerical feature, what we can do is that we can considerf3as target feature and reframe this as regresion problem where we try to predict for missing values.The above setup is identical to the original regression task, here we would build a model to usef1andf2to predict forf3. So instead of usingmean,medianetc. we can build a model to restore missing values which can help us solve the original problem efficiently.We have to be careful to not overfit when building such models.Check Train Test DistributionsWhenever we train a model we want to use it on a new sample, but what if the new sample comes from a different distribution compared to the data on which the model was trained on. When we deploy our solutions on production we want to be very careful of this change as our models would fail if there is a mismatch in train and test sets. We can pose this problem as an auxiliary task and create a new binary targety, where1represents whether row comes fromtestset and0represents whether it comes fromtrainset and then we train our model to predict whether a row comes fromtrainortestset if the performance ( e.g.AUCscore ) is high we can conclude that the train and test set come from different distributions. Ofcourse, we need to remove theoriginal targetfrom the analysis.In the above images you can see two different datasets, we want to verify whether these two come from same distributions or not.Consider the first example set as training set and second one as test set for this example.We create a new target calledis_testwhich denotes whether a row belongs to test set or not.Then we combine training and test set and train a model to predict whether a row comes from train or test set, if our model performs well then we know that these two datasets are from different distributions.We would still have to dig deep into looking at whether that's the case but the above method can help identifying which features are have drifted apart in train and test datasets. If you look at feature importance of the model that was used to separated train and test apart you can identify such features.Feature Selection using Null ImportanceOften we create many features to enhance the power of our ML model, but not all features are beneficial to model performance. There needs to be a way to identify
features which can introducenoiseto the model and hence must be removed. We can make use ofnull importanceto identify and remove such features.Use LightGBM model to train on all features and obtain feature importance ( gain ), call it gain_1For the same data shuffle the target variable and train model and obtain feature importance ( gain ), call it gain_2Calculate ratio = gain_2 / gain_1 and remove features with ratio > 3.Use the remaining features to train the model again.Note: The ratio threshold of3seems arbitrary here, but we can treat this ashyper-parameterand choose the value that gives us the best result on the CV.Dataframe representing feature importance of useful, useless model and their ratio.Feature ImportanceGradient Boosting Trees are very powerful methods when it comes to solving problems in industry and often are the first choice of every practitioner. We often to look at feature importance which different implementations of GBDT provide in help us understanding what features are useful. But during the inference when we have to predict for an unseen example, e.g. in a classification task if our model spits out0or1it would be useful to know which features were instrumental in identifying it as positive or negative class. This formulation can be extended to multi-classification and regression problems.This module implements thispaperIt currently supports XGBoost model and will soon support other implementations.We just pass our trained model and it converts the tree representation into a dataframe.- Then we calculate score for each node of the tree where we start off from the terminal node and then propagate it up to the root node.Installtask_substitution in on pypi:pip install task_substitutionFor an editable install, use the following:git clone https://github.com/numb3r33/task_substitution.git
pip install -e task_substitutionHow to useRecover Missing ValuesCurrently we only support missing value recovery for numerical features, we plan to extend support for other feature types as well. Also the model currently uses LightGBM model to recover missing values.from task_substitution.recover_missing import *
from sklearn.metrics import mean_squared_error

train = train.drop('original_target', axis=1)

model_args = {
          'objective': 'regression',
          'learning_rate': 0.1,
          'num_leaves': 31,
          'min_data_in_leaf': 100,
          'num_boost_round': 100,
          'verbosity': -1,
          'seed': 41
             }

split_args = {
    'test_size': .2,
    'random_state': 41
}

# target_fld: feature with missing values.
# cat_flds: categorical features in the original dataset.
# ignore_flds: features you want to ignore. ( these won't be used by LightGBM model to recover missing values)

rec = RecoverMissing(target_fld='f3',
                     cat_flds=[],
                     ignore_flds=['f2'],
                     perf_fn=lambda tr,pe: np.sqrt(mean_squared_error(tr, pe)),
                     split_args=split_args,
                     model_args=model_args
                    )

train_recovered = rec.run(train)Check train test distributionsWe use LightGBM model to predict whether a row comes from test or train distribution.import lightgbm as lgb
from task_substitution.train_test_similarity import *
from sklearn.metrics import roc_auc_score

train = train.drop('original_target', axis=1)

split_args = {'test_size': 0.2, 'random_state': 41}

model_args = {
    'num_boost_round': 100,
    'objective': 'binary',
    'learning_rate': 0.1,
    'num_leaves': 31,
    'nthread': -1,
    'verbosity': -1,
    'seed': 41
}

# cat_flds: categorical features in the original dataset.
# ignore_flds: features you want to ignore. ( these won't be used by LightGBM model )

tts = TrainTestSimilarity(cat_flds=[], 
                          ignore_flds=None,
                          perf_fn=roc_auc_score,
                          split_args=split_args, 
                          model_args=model_args)
tts.run(train, test)

# to get feature importance
fig, ax = plt.subplots(1, figsize=(16, 10)
lgb.plot_importance(tts.trained_model, ax=ax, max_num_features=5, importance_type='gain')Feature Selection using Null ImportanceWe use LightGBM model to remove features based on null importance.from sklearn.datasets import load_boston
data = load_boston()

X = pd.DataFrame(data['data'], columns=data['feature_names'])
y = pd.Series(data['target'])

model_args = {
    'num_boost_round': 300,
    'objective': 'regression',
    'learning_rate': 0.1,
    'num_leaves': 31,
    'nthread': -1,
    'verbosity': -1,
    'seed': 41
}

print(f'Feature list: {X.columns.tolist()}')
fs = FeatureSelection(model_args)
selected_features = fs.select_features(X, y)
print(f'Selected features: {selected_features}')

# ratio of feature importance of useless / useful model
print(fs.ratio_df)Feature Importance for GBDTMake your GBDT more explainable by calculating feature contribution that reveals the relationship between specific instance and related output. Now in addition to global feature importance we can breakdown our predictions and find out which features played positive or negative role during inference.x,y = make_regression(n_samples=1000,n_features=6,n_informative=3)
xtr, xval, ytr, yval = train_test_split(x, y, test_size=0.5, random_state=41)

model = xgb.XGBRegressor(objective='reg:squarederror', n_estimators=10, max_depth=4)
model.fit(xtr,ytr)

lfi     = LocalFeatureImportance(model)
scores  = lfi.propagate_scores()

si = pd.DataFrame(xval[5:6, :], columns=['f0', 'f1', 'f2', 'f3', 'f4', 'f5'])
fc = lfi.get_fi(scores, si)ContributingIf you want to contribute totask_substitutionplease refer tocontributions guidelines