Correlation Matrix for non-numeric features Currently, I have dataset with numeric as well non-numeric attributes. I am trying to remove the redundant features  in the dataset using R Programming Languages. Note: Non-numeric attributes cannot be turned into binary.

The Caret R package provides the findCorrelation which will analyze a correlation matrix of your data’s attributes report on attributes that can be removed. However, It only works numeric values of 'x'. I have been unable to find a package which does it for non-numeric attributes.

Is there a function in Caret R Package that does that for non-numeric attributes as well? If not, any method/package that would help me achieve the same?

There are measures of association for categorical variables. If you are looking at two ordinal variables you may use Spearman's correlation coefficient. There are also many measures for association for purely categorical variables, such as gender and race. Yule's Q and Crammer's v are popular choices.

two options come to mind looking at your question:

1) One-Hot-Encoding + Correlation:
One-Hot-Encoding takes your categorical features and replaces them by making a binary column out of each unique value of your categorical feature. Using this you would get a bunch of new numerical/binary features that you could then use for claculating the correlation between each bineary feature and your numerical ones.

2) Logistic Regression:
If you want to measure correlation between your categorical feature and a numeric one you could try calculating a logistic regression between both (having your categorical feature as target and the numeric one as your only input). The hypothesis behind this is, that assuming you have a high correlation between your categorical feature and a numeric one it should be possible to build a fairly good performing model for predicting your categorical feature using your numerical one as input.

Of course there are a bunch more ways to measure correlation between categorical and numerical values. They all however are based on some sort of distance measure you need to define for your categorical features.


Update machine learning model while retaining the prediction results exact for old data New to ML here. In our industry, we are looking for a type ML method/model that can be updated to accommodate new data points while keeping the prediction value of the historical data exactly the same. Of course a simple hash table would do, but are there other better alternatives.
First question is what is this type of method/model called in the ML world. Where could I find some good examples/explanations of such techniques?
Thanks

The safest way to keep the same predictions for your older data is versioning your trained models. It is, keeping the generated model artifact (in python could be pickle file, h5 file, etc) to make sure that you can use it getting the same results as you say, and generating new models (so new artifacts) via retraining when new data come in.
The usual approach is deciding an evaluation metric for your ML model, and based on it, when new data come in, you retrain your models (the frequency for this is something to decide on your team) and when the metric value is better (enough) than the latest model version, you can promote this new model for use in production. Theere are quite goos frameworks for models versioning, as MLflow.
Keep in mind that transfer learning does not guarantee predictions on older data are going to be same, since a new model is trained based on another model, but a different model is eventually created.

The wording of the question is a bit vague, but I believe the machine learning term you are looking for is transfer learning, which is essentially recycling a previously trained model while adding new data. This is a common technique for data scientists to take advantage of existing models (oftne created by groups with much more data and resources, such as Google) and customizing it for their own specific use-case. For example, taking an existing, general NLP model and training it to work better on StackExchange comments.
Depending on the problem, there are other concepts like lifelong learning and continual learning, but transfer learning is a better starting point for someone new to ML.


Many separation line using RBF kernel in SVM Below is my code, it take a range of a number, creates a new column label  that contains either -1  or 1.

In case the number is higher than 14000 , we label it with -1 (outlier)
In case the number is lower than 14000 , we label it with 1 (normal)

## Here I just import all the libraries and import the column with my dataset
## Yes, I am trying to find anomalies using only the data from one column

df['label'] = [-1 if x >= 14000 else 1 for x in df['data_numbers']]  #What I explained above

data = df.drop('label',axis=1)
target = df['label']
outliers = df[df['label']==-1]

outliers = outliers.drop('label',axis=1)

from sklearn.model_selection import train_test_split
train_data, test_data, train_target, test_target = train_test_split(data, target, train_size = 0.8)
train_data.shape

nu = outliers.shape[0] / target.shape[0]
print("nu", nu)

model = svm.OneClassSVM(nu=nu, kernel='rbf', gamma=0.00005)
model.fit(train_data)

from sklearn import metrics
preds = model.predict(train_data)
targs = train_target
print("accuracy: ", metrics.accuracy_score(targs, preds))
print("precision: ", metrics.precision_score(targs, preds))
print("recall: ", metrics.recall_score(targs, preds))
print("f1: ", metrics.f1_score(targs, preds))
print("area under curve (auc): ", metrics.roc_auc_score(targs, preds))
train_preds = preds

preds = model.predict(test_data)
targs = test_target
print("accuracy: ", metrics.accuracy_score(targs, preds))
print("precision: ", metrics.precision_score(targs, preds))
print("recall: ", metrics.recall_score(targs, preds))
print("f1: ", metrics.f1_score(targs, preds))
print("area under curve (auc): ", metrics.roc_auc_score(targs, preds))
test_preds = preds

from mlxtend.plotting import plot_decision_regions                                 # as rbf svm is used hence lot's of  decision boundaries are drawn unlike one in linear SVM
# the top one central points with blue quares are outlietrs while at the bottom they are orangy triangles(normal values)
plot_decision_regions(np.array(train_data), np.array(train_target), model)
plt.show()

Output from training data

accuracy:  0.9050484526414505
precision:  0.9974137931034482
recall:  0.907095256762054
f1:  0.9501129131595154
area under curve (auc):  0.5876939698444417

Output from test data

accuracy:  0.9043451078462019
precision:  1.0
recall:  0.9040752351097179
f1:  0.9496213368455713
area under curve (auc):  0.9520376175548589

My graph seems to be having so many sepearation lines, I was thinking I would only be getting one that differentiates between the outliers and the normal data.

The OneClassSVM is an unsupervised algorithm that is supposed to learn the normal data distribution. This means that the algorithm will model the boundaries of areas of high likelihood for your data points to be drawn from.

Even though you know which points are outliers, the SVM will only know that there is roughly a proportion of outliers of nu, therefore the 14000 boundary is not intuitive at all in terms of distribution.

What the SVM will do that explains your multiple lines is:

If you have a hole in what you consider the "normal zone" - let's say as an example you have no data points between 2560 and 2985 - the SVM can decide that this is an area of low likelihood for your distribution and therefore build two vectors to exclude it from the learned normal distribution.
At the opposite, if you have several points clustered together above 14000, since the SVM has no clue this is supposed to be the anomalous zone, it can detect a zone of high likelihood for the data points to be drawn from, and build two vectors to include it in the learned normal distribution.

Now regarding your metrics:

You probably have an imbalanced dataset since we are talking about anomaly detection (otherwise it makes no sense!), so you should not use global accuracy here because it is misleading.
There is one thing that should alert you: The train AUC is very low, while the test AUC very high and this is not normal. So your algorithm poorly learns the target normal distribution.

The high test AUC can be explained, but first let's use some notations:

The real normal data distribution will be called *the target normal distribution", in the sense that you want to learn it. It is the interval [0;14000[
The real anomalous area (cannot talk about a distribution in this setting) will be called "the target anomaly area". It is the interval [14000;+infinity[
The learned normal distribution will be called the "learned normal distribution", obviously.
The learned anomalous area will be called the "learned anomalous area"

So the learned distribution consists in small clusters around areas of higher training data density. Since you (supposedly) have a few outliers and a lot of normal points, the learned distribution still intersects well with the target normal distribution, but not with the target anomalous area, where only a few areas will belong to the learned normal distribution. You can verify this on your graph: there are only a few vectors in the target anomalous area.

Now when you apply the algorithm on the test set, it is not likely that a point from the test set in the target anomalous area falls into the intersection of the learned normal distribution and the target anomalous area, because this intersection is very small. But this is only due to the small number of data points in this anomalous area, not to a successful learning process. Hence these points will still be labeled as anomalies and you get a misleading high AUC score.

At the contrary, the target normal distribution and the learned normal distribution intersects correctly, because there was more data points to learn from. So most of the normal points in the test set are classified correctly.

To sum up, your algorithm learns a completely different distribution from what you excpected. You can try to use a higher gamma value to smooth the kernels. but you will probably not avoid the classification of the areas of higher density above 14000 as normal data points. So watch out for the train AUC!

It would be very interesting to investigate these statements with graphs, you could as an example plot the [14000; +infinity[ zone, highlight the decision function boundaries and plot train and test set data points. You should get confirmation of what I am saying above (I will do it later and add the graphs if I find some time!)

If you know the decision boundary(14000) then why you need an ML algorithm? You can apply an if the condition on that. ML algorithms are developed for finding a decision boundary? If you really want to experiment on this dataset then you have to know what type of classification you are doing?
From the artificially labeled dataset, I can see you have two classes(1, -1). But you are using OneClassSVM. That's the mistake! Just remove OneClassSVM and use normal linear svm. And no need to use 'rbf'. 'rbf' is for nonlinear separation. Your classification boundary is linear. It just a line going through 14000.

from sklearn import svm
clf = svm.SVC()
clf.fit(train_data, train_target)

Note: If you want to do outlier detection then don't label the data by (< 14000) condition. Allow the OneClassSVM to find the outlier by looking training features only.


Early stopping on validation loss or on accuracy? I am currently training a neural network and I cannot decide which to use to implement my Early Stopping criteria: validation loss or a metrics like accuracy/f1score/auc/whatever calculated on the validation set.

In my research, I came upon articles defending both standpoints. Keras seems to default to the validation loss but I have also come across convincing answers for the opposite approach (e.g. here).

Anyone has directions on when to use preferably the validation loss and when to use a specific metric?

TLDR; Monitor the loss rather than the accuracy

I will answer my own question since I think that the answers received missed the point and someone might have the same problem one day.

First, let me quickly clarify that using early stopping is perfectly normal when training neural networks (see the relevant sections in Goodfellow et al's Deep Learning book, most DL papers, and the documentation for keras' EarlyStopping callback).

Now, regarding the quantity to monitor: prefer the loss to the accuracy. Why?
The loss quantify how certain the model is about a prediction (basically having a value close to 1 in the right class and close to 0 in the other classes). The accuracy merely account for the number of correct predictions. Similarly, any metrics using hard predictions rather than probabilities have the same problem.

Obviously, whatever metrics you end up choosing, it has to be calculated on a validation set and not a training set (otherwise, you are completely missing the point of using EarlyStopping in the first place)

I am currently training a neural network and I cannot decide which to use to implement my Early Stopping criteria: validation loss or a metrics like accuracy/f1score/auc/whatever calculated on the validation set.

If you are training a deep network, I highly recommend you not to use early stop. In deep learning, it is not very customary. Instead, you can employ other techniques like drop out for generalizing well. If you insist on that, choosing criterion depends on your task. If you have unbalanced data, you have to employ F1 score and evaluate it on your cross-validation data. If you have balanced data, try to use accuracy on your cross-validation data. Other techniques highly depend on your task.

I highly encourage you to find a model which fits your data very well and employ drop out after that. This is the most customary thing people use for deep models.


How to replace NA values with another value in factors in R? I have a factor variable in my data frame with  values where in the original CSV "NA" was intended to mean simply "None", not missing data. Hence I want replace every  value in the given column with "None" factor value. I tried this:

DF$col[is.na(DF$col)] <- "None"

but this throws the following error:

Warning message:
In `[<-.factor`(`*tmp*`, is.na(DF$col), value = c(NA, NA,  :
  invalid factor level, NA generated

I guess this is because originally there is no "None" factor level in the column, but is it the true reason? If so, how could I add a new "None" level to the factor?

(In case you would ask why didn't I convert NAs into "None" in the read.csv phase: in other columns NA really does mean missing data).

You can use this function :

forcats::fct_explicit_na

library(forcats)
fct_explicit_na(DF$col, na_level = "None")

Usage

It can be used within the mutate function and piped to edit DF directly:

library(tidyverse) # for tidy data packages, automatically loads dplyr
library(magrittr) # for piping
DF %<>% mutate(cols = fct_explicit_na(col, na_level = "None"))

Note that "col" needs to be a factor for this to work.

change data type

d=as.matrix(d)
d[is.na(d)] <-"None"
d=as.data.frame(d)


Ensemble Model vs Normal model If I get 95+ % accuracy in normal models, should I still consider Ensemble models? Why should I choose Ensemble models over normal models?

It is a problem specific and data specific problem. For one problem, 95% accuracy may be good, for some other it might not be very good and there might still be room for improvements. Likewise, it might happen that the problem is well defined, your architecture is good but due to a lot of noise in the data, you can't do better than 95%. In that case, we will say the results are good.

Ensembles: Ensembling is a general term for combining many classifiers by averaging or voting. It is a form of meta-learning in that it focuses on how to merge results of arbitrary underlying classifiers. Generally, ensembling improves the final results but again, not necessarily. If you are ensembling two highly correlated models, your ensembled model will give almost the same result as your stand-alone models.

Firstly, welcome to the site!

When do we use Ensemble model?

when there are 2 models which perform moderately then we combine their results to get a model which performs better. In your scenario you already have a model which gives you good results what is the point of implementing Ensemble Models?

As @Tagoma said, it depends on your data and your goal. For example, you are trying to predict stock rates, every 0.01% matters. In such scenarios you need to use complex algorithms to maintain balance on that slim line i.e., not to over-fit, not to over-train and just predict.

Measures to check if your model is over-trained is by giving some random data and see how it performs, add some noise in the training data.

One more important thing to do is, to check for the Predictor Importance and see if there is any highly correlated feature with the target variable. For example, you are planning to predict age and if you have DOB as a feature in that yes of course you would predict with 99.99% accuracy but that is not what we use ML for.

If all of them are satisfied and achieving that accuracy then it means that your model's performance is good.

Finally, Implementation of Ensemble is dependent on your business problem and your understanding on business.


R problem with fraction numerals Why do I have a non-zero result in the following calculation:

  0.11-0.1-0.01
  [1] -5.20417e-18

Seems to be an easy and akward question, but I will be very greatful for any help!)

You can multiply with (X==)10/100/1000/... to size of precision you want. After performing multiplication to both numbers, Do your operation(-, /, ...) You will get the result.


Can my neural network learn conditional rules when classifying? I'm concerned that I'm attempting the impossible with my neural network. This is the scenario:
I have a 2D square world. In it, I create five circles of random size and position. I then classify one of them as the correct answer, based on the following rules:

If any circle's radius is > THRESHOLD, I choose the largest circle
Otherwise, I choose the circle with the origin nearest the center

I send the inputs as serial coordinates, like this: [X0, Y0, RADIUS0, X1, Y1, RADIUS1, ...].
The output is a one-hot array, e.g. [0, 0, 1, 0, 0].
I've modeled this in TensorFlow without success. My best scoring result appears to always choose the largest circle, ignoring the else clause of the arbitrary rule.
Am I fundamentally misunderstanding the capabilities of neural networks? I've tried many (many) different configurations (layer counts, node counts, activation functions ... you name it). All of my networks have been feed-forward, so far.
Thanks in advance for any insight!

Here are some details of my network and data:

I have tried with up to 500k cases. I separate 10% for generalization checks after training, and train on the remaining 90% with a 50/50 validation split.
I've tried with the test data weighted 75% toward rule A, 50/50, and 75% toward ruleB.
I've tried 0-10 hidden layers, and neuron counts from 2 to 256 (each hidden layer gets the same number of neurons).
I change the number epochs as time allows, but generally it's 10-100. My longest runs have been several hours (with giant case numbers, and dropouts to prevent overfitting).
I've used batch sizes of 1-50.
I've tried learning rates of 0.0001 - 0.1.
I'm currently using ReLU activation, initializing bias to const(0.1) and kernel w/ heNormal. I have tried several other approaches for all three.
I standardize the inputs to center on zero w/ variance of one.
The loss function is categoricalCrossentropy.
The optimizer is Adam.

Centering the data on zero and scaling to variance of one helps with a lot of classification problems but in this case it would remove information that's needed to solve your problem as I understand it.
Another possible problem is the loss function, which you don't mention at all in your question. I would suggest something that stays fairly high when your neural network is only learning one of the two rules.

Yes, the Universal Approximation Theorem states that a neural network can learn any function in $R^n$ with one hidden layer and a finite number of neurons with a non-linear activation function. There are many things that can go wrong with training a network, for one, have you tried graphing its performance over time and seeing if it is converging?


Adjusted R-squared is too high (=1) in Linear Model I built a Linear model which has an adjusted r-squared value of 1. I understand that this is a near perfect number. Upon further investigation, I found that one of the 96 independent variables in the dataset is highly correlated with the dependent variable. This is also a variable which I would like to keep (and not drop). Are there any additional steps that I should undertake to handle this?

Sample df reproducing the situation above:

df1 <- data.frame("y" =   c(0.0166 , -0.2380 , -0.3192 , -0.2774 ,  9.3148 , 0.3142) ,
          "x1" =  c(0.0103 , -0.2347 ,  -0.3182 , -0.2793 ,  9.4638  , 0.3297) ,
          "x2" = c( -0.1838 , -0.2458 , -0.2581 ,-0.2533 , 6.7566 ,-0.0835) ,
          "x3" = c(0.3426 ,-0.0543 ,-0.4512 ,-0.0543, 10.4637 , 0.3426) ,
          "x4" = c(-0.161 , -0.270 ,-0.318, -0.280 , 8.279 , 0.169))
df1
df1_lm <- lm(y~. , data = df1)
summary(df1_lm)

I'd strongly disagree with Viktor an adjusted r² of 1 is not "luck to have a good model" it is a clear indicator of data leakage and a model that will absolutely overfit on new data!
To mediate this you need to understand your data and domain better (which means we need more infos to help).
But generally try to answer the following questions:

In any case I have not measured Y would I also be unable to measure X (the good variable)?

Is X a derivative value of Y? E.g. if you try to predict height in meters it does not make sense to include height in cm, you'd simply calculate it.

What is my use case? Do you want to explain relations / impact, do you want to predict something on new unseen data, etc.? Depending on the use case you have to deal with it in different ways.

It does not make sense however to keep this variable in your model because it invalidates the need for the model in the first place.
Either you could go ahead simply calculating Y from X and ignore all other variables (in case it is a simply unit conversion or a deterministic classifier like age > 18y --> adult) or you have to remove X to actually learn something about all the other variables (in case you want to explore impact / relation of other variables).
Also keep in mind that unless the linear regression of Y and the single good X is also r² = 1 you might have to remove or cluster two variables!

You are lucky to have a good model, no need to handle this if there is no information leakage from y to your x1 variable.

Checking the data you provided you may want to eliminate variables that are not significant though.


Random Forest Classifier - KFold CV Tunes Very Deep Trees --> Overfitting? I'm tuning a random forest in python and am wondering if/why my model is overfit. The dataset is described below:

1700 Positive Cases / 54000 total cases ~ 3.2% (unbalanced)
50 Numerical Features,~450 label/hot encoded features (post data
reduction)
10Fold CV using 85% of data, with 15% hold out for final test
Classification Metrics = AUC or F1 (as data is imbalanced)

The results I get tend to suggest using very deep trees i.e depth 18 with no restriction on number of samples per split = 2(default). In this case, Train AUC was 99.9% , Max Test AUC was 84%. My scores are also almost monotonically increasing in max depth of trees. Given the results and how deep the trees are - I suspect the model is overfit? If this is the case then why would I not observe some sort of out of sample reduction in AUC as depth and min_samples_split typically constrain the random forest? Or have I overlooked anything in tuning?

My ranges in CV Grid Search are more or less:

n_estimates : range(100,1000,by=100)
max_features : {sqrt(p),0.3,0.4,0.5}
max_depth : range(2,20,by=1)
min_samples_split : range(2,50,by=1)
class_weights : {balanced,None}

Thanks

There are signs of overfitting the data when the model's performance drops on the test set considerably.

Good feature engineering can also improves your model's performance. Considering that you have an imbalanced data set, you can look at resampling techniques for your data set such as random over/under-sampling and SMOTE. You can also extract feature importances from ensemble learners or regularized regression models like LASSO to see which features are contributing the most weight in your model's predictions. You could also apply various statistical tests such as ANOVA or chi-square square tests to gain further insights of your data. There is lots to do here.

You can also reconsider using the ROC AUC metric and choose something better tuned to a class imbalance in your data set. The ROC curve plots the True Positive Rates (TPR) on the x-axis and the False Positive Rates(FPR) on the y-axis whose formulas respectively correspond to $$TPR=\frac{TP}{TP+FN}$$ and $$FPR=\frac{FP}{FP+TP}$$ Since TPR is based only on the TP instances, the ROC curve won't measure the effects of negative instances. The AUC does does not consider the class imbalance when evaluating overall model performance, so we cannot trust it to weigh on the underrepresented class.

One way to resolve this is by using the Matthews Correlation Coefficient since it takes into account true and false positives and negatives. This helps represent the minority class a lot more during your model evaluation.

You could also look into random grid searched for your hyperparameter tuning; this method uses a fixed number of parameter settings sampled from your specified distribution. This allows you to get a general neighborhood of your parameter space where you can use a brute force grid search in a shorter parameter space. Then you can use cross validation to evaluate your model as you normally would.

This is just a handful of things to consider, there are many other things that you may find are suitable for your problem.

Random Forests don't overfit, the more depth you add, the more accuracy and less performance you will get.


How to automatically verify official documents? I am new to machine learning and data science. I apologise if the question seems very basic. I have a requirement where I need to verify information submitted via a form with the corresponding official document. My approach till now has been to use google vision to extract text and use regex to extract necessary fields and compare with the form information. This is not always reliable because of the image quality and vision captures the noise as well.

Previously, I was thinking of just comparing each data file and search it in the text extracted and provide a metric of certainty.

I talked to one of my colleagues and he suggested using some for of supervised learning algorithm to process the documents, so that they automatically extract the key fields.

I want to ask would this method be simpler than my current approach. I am also worried about the scalability of my approach in event of minor changes in document formats.

I am looking for links for some articles or books related to this and answer to why using my own model will be better than just searching each word in a text.

Edit: The data can be imagined as a visiting card which contains the name of person, his office address, his contact number and the company's name. The problem here is different visiting cards can have different formats. Moreover the information is repeated twice: Once in English and the second in another language. For example , NAME : JANE DOE is followed by नाम : जेन डोए in next line. I only need the English name. The number of formats are constant but are high. Moreover, the address is not read together by Google Vision in most cases and using regex can become too complicated and case specific. I need to verify employee identity cards as proof that they are working for the company they claim to.

Please refer the below link which is related to Machine learning in document analysis and recognition.

https://www.researchgate.net/publication/242506468_Machine_Learning_in_Document_Analysis_and_Recognition


GAN: Discriminator converges, generator learns almost nothing In my GAN, the discriminator loss goes down steadily, while the generator loss oscillates / does not converge.

I suspect this is due to the vanishing gradient problem. Theory: as the discriminator loss doesn't start out big to begin with, the generator will never be backpropagated small gradients into, and therefore it doesn't learn to generate better images.

What are some remedies to resolve this and make the generator actually work?

This is one of the bigger problems with GANs, they are usually hard to train.

This is an active area of research and it does not seem like this problem has been solved. Without additional information or code, it is hard to help, but usually different loss functions are one way to mitigate the problem.


Find the furthest pages of a website I would like to find the two furthest pages of a big websites (+5millions pages).

(It's the degree of separation, the most links between the two pages)

This is a classic "graph database" question. I would do the following steps:-

Traverse all the pages using a webcrawler, or if you can interrogate the website server directly that may be qukcker.
Ensure you keep all links between pages, for later use.
Put the web pages into a neo4j database with connections between the pages

This will allow you to be able to do the query of pages with the maximum number of steps, or pages most frequently linked to, etc.

Assume you have the data in a table containing a row for each link, storing the PAGE and the linked page (LINK_TO).

select * from page_link;

PAGE LINK_TO
---- -------
A    A
A    B
A    B
A    C
B    C
B    C
B    C
C    A

Than simple group on the two columns and count the occurences.
This gives the number of links between any two linked pages, so you can see the most links between the two pages - if this is realy what you want...

select PAGE, LINK_TO, count(*)
from page_link
group by PAGE, LINK_TO order by 3 desc;

PAGE LINK_TO   COUNT(*)
---- ------- ----------
B    C                3
A    B                2
A    C                1
A    A                1
C    A                1

A few milions of records should not be a problem for any database of your chioce.


How to set class weights for imbalanced classes in Keras? I know that there is a possibility in Keras with the class_weights parameter dictionary at fitting, but I couldn't find any example. Would somebody so kind to provide one?

By the way, in this case the appropriate praxis is simply to weight up the minority class proportionally to its underrepresentation?

You could simply implement the class_weight from sklearn:

Let's import the module first

from sklearn.utils import class_weight

In order to calculate the class weight do the following

class_weights = class_weight.compute_class_weight('balanced',
                                                 np.unique(y_train),
                                                 y_train)

Thirdly and lastly add it to the model fitting

model.fit(X_train, y_train, class_weight=class_weights)

Attention: I edited this post and changed the variable name from class_weight to class_weights in order to not to overwrite the imported module. Adjust accordingly when copying code from the comments.

I found the following example of coding up class weights in the loss function using the minist dataset. See link here.
def w_categorical_crossentropy(y_true, y_pred, weights):
    nb_cl = len(weights)
    final_mask = K.zeros_like(y_pred[:, 0])
    y_pred_max = K.max(y_pred, axis=1)
    y_pred_max = K.reshape(y_pred_max, (K.shape(y_pred)[0], 1))
    y_pred_max_mat = K.equal(y_pred, y_pred_max)
    for c_p, c_t in product(range(nb_cl), range(nb_cl)):
        final_mask += (weights[c_t, c_p] * y_pred_max_mat[:, c_p] * y_true[:, c_t])
    return K.categorical_crossentropy(y_pred, y_true) * final_mask


PySpark: How do I specify dropna axis in PySpark transformation? I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument axis = 'columns' in dropna(). Here an example in a GitHub post.

How do I do this in PySpark ? dropna() is available as a transformation in PySpark, however axis is not an available keyword.

Note: I do not want to transpose my dataframe for this to work.

How would I drop the furniture column from this dataframe ?

data_2 = { 'furniture': [np.NaN ,np.NaN ,np.NaN], 'myid': ['1-12', '0-11', '2-12'], 'clothing': ["pants", "shoes", "socks"]}

df_1 = pd.DataFrame(data_2)
ddf_1 = spark.createDataFrame(df_1)
ddf_1.show()

I  know this is a bit late, but I struggled with this also. This is my attempt at removing null columns from a Spark Dataframe.
from pyspark.sql.functions import when, isnull

colsthatarenull = df.select([(when(isnull(c), c)).alias(c) for c in df.columns]).first().asDict()
namesofnullcols = {key:val for key, val in colsthatarenull.items() if val != None}.values()
df = df.drop(*namesofnullcols)

You should be able to use the column name like:

df_1 = df_1.drop('furniture')


Why does changing the cluster number change the plot in Kmeans? This might be a dumb questions but I can't find the answer to it. I don't have the perfect mathematical understanding of kmeans, so apologies if it is.

I'm just wondering why I see a different plot when I change the number of clusters in a kmeans plot?

Here's the code that I'm using:

set.seed(1)
k <- kmeans(data, centers = x)
plotcluster(data, k$cluster)

I vary x to see how the plot looks like. Below are the results for x = 3 and x = 4. Apologies for the poor formatting.

I'm wondering why both plots look different if I'm only varrying the number of clusters. Is it because the principle components that are being show, dc1 and dc2, are different as you change x, so as to maximize the variance displayed?

Another quick question is - can you determine the cluster # by how "neat" the clustering looks in the plot? I know there are various methods of doing it, I'm just wondering if the plot is indicative of good/bad clustering as well.

any help is appreciated!

plotcluster changes the projection based on the result that you give. See the documentation for more details.

Because of that you cannot compare the two plots.

why I see a different plot when I change the number of clusters in a
  kmeans plot?

The clusterization of each observation, in k-Means clsutering, depends from the number of clusters (centroids) that you choose. Their centroids self-adjust by iterating the algorithm, so the position of each of them depends from the position of others. That is why introducing a new centroid shifts the others to a new equilibrium.

--

  can you determine the cluster # by how "neat" the clustering looks in
  the plot?

The common way of choosing their number is using the elbow method. It is based on repeating the algorithm multiple times, each time with a different k, checking the amount of information/error that each clustering returns. By looking at the plot of these iterations you can choose the amount of clusters that satisfies you.


Predicting High-School test scores after a disciplinary action I'm somewhat new to machine learning and have learned to apply many of the basic regression and classification methods using python and various packages.  However, approaching this problem has me stumped.  To illustrate the problem, I created a fictitious scenario where a guidance counselor wants to predict test scores for a student after disciplinary action.  Suppose they have data available like the mock-up below:

Column definition:
Student - Student Identification #
Gender - Male/Female
Age - Current Age
Athlete - Sport that student plays (0-None, 1-Basketball, 2-Football, 3-Soccer)
Online - Student takes classes online (0-No, 1-Yes)
Before_Disciplinary_Action_Scores - Sequence of their last n test scores prior to discipline (in date order)
Discplinary_Action - Action Counselor Took (0-None, 1-Assigned Tutor, 2-Guardian Meeting, 3-Study Program, 4-Ineligible for game)
After_Disciplinary_Action_Scores - Sequence of their next n test scores after discipline (in date order)

Assume the fictitious school system is quite large and there are about 80k records in total.  All students have a total of 12 test scores from Before/After discipline action, but the number of test scores varies based on when the disciplinary action was given.  For simplicity, you can assume the scores are given on a weekly basis for a single course.

I can compute mean before/after scores and create a fairly good classification model, but I'd like to take it a step further and predict the post-discipline scores.

I've tried using Prophet and LSTM models to predict the after scores from a time series approach with poor results and the varying number of scores makes it difficult.  I can see that the features around Athlete, Transfer, and Online are important components and I tried adding them as regressors, but that too has failed.  I appreciate any guidance you can give.

If your data is correct then i see a factor of gender also. I hope that is taken into account and also take Disp_action as categorical variable (one hot encoding).
Also, try to club pre and post action scores in one fields as output variable. Another field can simply tell if the output score is pre action or post action.
This feature engineering, I hope will give better results.


What is the difference between parameters & cooficients in Machine learning? are both terms interchangeable?
I'm kinda new to machine learning field and very confused about these terms in machine learning perspective.

When talking about coefficients, we usually refer to some fixed numbers or constants, e.g. learning rate coefficient is 0.001 and it will not be changed during training. Parameters are usually changing, e.g. model is training to adjust right parameters. However, there are no strict borders between these two notions.


Machine Learning Perceptron Algorithm I'm studying Machine Learning using Sebastian Raschka's book.

Wonder if someone could please help me to confirm if I have the following steps correct if I apply Perceptron Algorithm to Iris dataset as shown in the table below.
What do I set $\eta$ to?
What do I set $θ$ to? I randomly set it to 2.

Perceptron rule - summarized by the following steps:

Initialize the weights to 0 or small random numbers.
For each training sample $x(i)$ perform the following steps:

Compute the output value $\hat{y}$ .
Update the weights.

\begin{array}{c|c|c|}
 & \text{sepal_length, $X_1$ } & \text{sepal_width, $X_2$} & \text{petal_length, $X_3$} & \text{petal_width, $X_4$} & \text{species, $Y$}\\ \hline
\text{Row 0} & 5 .1 & 3.5 &1.4 &0.2 & setosa \\ \hline
\text{Row 1} & 4.9 & 3.0 & 1.4 & 0.2 & setosa \\\hline
\text{Row 2} & 4.7 & 3.0 & 1.4 & 0.2 & setosa \\\hline
\text{Row 3} & 4.6 & 3.1 & 1.5 & 0.2 & setosa \\\hline
\text{Row 4} & 5.0 & 3.6 & 1.4 & 0.2 & setosa \\\hline
\end{array}

set: $θ = 2$

set: $virginica = -1,$ $\>$ $setosa = 1$

let's say if $z \ge  θ,$ $\>$ $θ(z) = 1$,  $\>$ $θ(z) = -1$ if otherwise

Initialize weight vector: $W$ = [ $w_1$ = 0.1, $w_2$ = 0.2, $w_3$ = 0.3, $w_4$ = 0.4]

For observation row 0:

$z = w_1*x_1 + w_2*x_2 + w_3*x_3 + w_4*x_4$
$= (0.1* 5.1) + (0.2*3.5) + (0.3*1.4) + (0.4*0.2)$
$= 1.71$

Since $z = 1.71 \lt θ = 2$, $\>$ then $θ(z) = -1$

As for row 0, the perceptron algorithm incorrectly predict $virginica = -1,$, hence the weights would be adjusted.

$w_j:= w_j + \Delta w_j$

$\Delta w_j = \eta(y^{(i)} - \hat{y}^{(i)}) x_j^{(i)}$

Hence,

$\Delta w_1 = \eta(1-(-1))5.1 = 10.2\eta$

$\Delta w_2 = \eta(1-(-1))3.5 = 7.0\eta$

$\Delta w_3 = \eta(1-(-1))1.4 = 2.8\eta$

$\Delta w_3 = \eta(1-(-1))0.2= 0.4\eta$

Update weight vector: $W$ = [ $w_1 = 0.1 + 10.2\eta$, $w_2 = 0.2 + 7.0\eta$, $w_3 = 0.3 + 2.8\eta$, $w_4 = 0.4 +0.4\eta$]

Using the updated weight to Row 1 and repeat the process above.

OK WHAT I understood from your understanding is below:-

1.What do I set η to?
Answer:**It is learning rate. As usual everyone uses **η=0.01. Learning rate is crucial because if you set bigger values when updating weights the it will reach to a local minima and never reach to global minima. Now if you set it low then it will gradually went to global minima. This iris dataset problem is so trivial that it takes no time to reach global minima but for a complex dataset you will need another factor such as decay rate. After a while, when training you will find the accuracy isn't increasing then the neural net suppose that it reached global minima now if you set decay rate to learning rate like 0.01* decay rate then learning rate will be much less and then it will TRULY converges to global minima as the gradient will be minimized in a small amount.

2.Z is the label for your dataset.You should have used one hot encoding. I think rather than writting everything here i would suggest following links.
Please go to there and then pick up a paper and pen.do the math.Bang.

How Neural Network learns

And obviously check 3Blue1Brown youtube channel's playlist on Neural Net.

Welcome to Deep Learning.Best of luck.


Over fitting in Transfer Learning with small dataset I am using Transfer Learning to perform image classification.
Base model used : Resnet50 using ImageNet dataset
class_1 and class_2 are the classes each having 1000 samples each (small dataset). And the dataset is not similar to ImageNet dataset.
Number of FC layers used here are 3 with [1024, 512, 256].
I have used a drop out of 0.5 to reduce over-fitting.
When I trained the model with 100 epochs, I could clearly see the model over-fits with training accuracy of 0.9985 and testing accuracy of 0.875.
Is the number of FC layers used is too many which is causing this over-fit problem?
How can I make the model more generalised?
The code used is as given below:
from keras.applications.resnet50 import ResNet50, preprocess_input
from keras.preprocessing.image import ImageDataGenerator
from keras.layers import Dense, Activation, Flatten, Dropout
from keras.models import Sequential, Model
from keras.optimizers import SGD, Adam
from keras.callbacks import TensorBoard
import keras
import matplotlib.pyplot as plt

HEIGHT = 300
WIDTH = 300
TRAIN_DIR = "/home/ubuntu/dataset/training_set/"
TEST_DIR = "/home/ubuntu/dataset/test_set/"
BATCH_SIZE = 8
class_list = ["class_1", "class_2"]
FC_LAYERS = [1024, 512, 256]
dropout = 0.5
NUM_EPOCHS = 100
BATCH_SIZE = 8

def build_finetune_model(base_model, dropout, fc_layers, num_classes):
    for layer in base_model.layers:
        layer.trainable = False

    x = base_model.output
    x = Flatten()(x)
    for fc in fc_layers:
        print(fc)
        x = Dense(fc, activation='relu')(x)
        x = Dropout(dropout)(x)
    preditions = Dense(num_classes, activation='softmax')(x)
    finetune_model = Model(inputs = base_model.input, outputs = preditions)
    return finetune_model

base_model = ResNet50(weights = 'imagenet',
                       include_top = False,
                       input_shape = (HEIGHT, WIDTH, 3))

train_datagen = ImageDataGenerator(preprocessing_function = preprocess_input,
                                   rotation_range = 90,
                                   horizontal_flip = True,
                                   vertical_flip = False)

test_datagen = ImageDataGenerator(preprocessing_function = preprocess_input,
                                  rotation_range = 90,
                                  horizontal_flip = True,
                                  vertical_flip = False)

train_generator = train_datagen.flow_from_directory(TRAIN_DIR,
                                                    target_size = (HEIGHT, WIDTH),
                                                    batch_size = BATCH_SIZE)

test_generator = test_datagen.flow_from_directory(TEST_DIR,
                                                  target_size = (HEIGHT, WIDTH),
                                                  batch_size = BATCH_SIZE)

finetune_model = build_finetune_model(base_model,
                                      dropout = dropout,
                                      fc_layers = FC_LAYERS,
                                      num_classes = len(class_list))

adam = Adam(lr = 0.00001)
finetune_model.compile(adam, loss="categorical_crossentropy", metrics=["accuracy"])

filepath = "./checkpoints" + "RestNet50" + "_model_weights.h5"
checkpoint = keras.callbacks.ModelCheckpoint(filepath, monitor = ["acc"], verbose= 1, mode = "max")
cb=TensorBoard(log_dir=("/home/ubuntu/"))
callbacks_list = [checkpoint, cb]

print(train_generator.class_indices)

history = finetune_model.fit_generator(generator = train_generator, epochs = NUM_EPOCHS, steps_per_epoch = 100,
                                       shuffle = True, callbacks=callbacks_list, validation_data = test_generator)

Update :

Weight file generated from the model after training is 2.7 GB. Is it normal considering the complexity of the model?

How would I select the steps_per_epoch value? Is there any standard?

First of all:

I think you should reduce the number of FC layers and number of nodes of FC layers, for example, one FC with 256 or 512, or 2 FC with 256 and 512. Try this.

Try to make your batch size 30, and decrease number of epochs to nearly 10 or 20. 100 epochs are too many for your small size dataset.

Secondly, there is more than one way to reduce overfitting:
1- Enlarge your data set by using augmentation techniques such as flip, scale, etc.
2- Using regularization techniques like dropout (you already did it), but you can play with dropout rate. Try more or less than 0.5.
3- One of the good techniques in your case is to do early stopping. In any epoch when you see that the model goes to overfit, stop it.
4- Using cross-validation to train/test your model.
and many more.

For more, see here, or here, or here.

Feel free to ask any further questions.

Be careful with Keras Batch Normalization.  You can try this code:

K.set_learning_phase(0)
input_tensor = Input(shape(img_size, img_size, 3))
base_model = ResNet50(input_tensor=input_tensor, include_top=False, weights="imagenet", pooling="avg")
x = base_model.output
#Define your own top layers
K.set_learning_phase(1)
x = Dense()
...
x = Dense()
model = Model(input_tensor, x)
for layer  in base_model.layers:
    layer.trainable = False

Or you can try to unfreeze the last few convolution layers, that might help. But still, be careful with Batch Normalization. There are many discussions available about this problem with keras's transfer learning.


predicting probability distribution for time series I have time series of several variables. Just in one specific case one variable is linear combination of the rest.

I want to predict probability distribution (that is not only best estimate but estimates with probability of that happening) of future value of variables. I want to see when probability of small interval of possible outcome is high.

A priory I don't know rules of the game how variables evolve and inter-depend.

What is the tool to best do such prediction and how easy is it? Will scikit-learn do? Maybe neural networks?

ADDED based on answer:

As I've understood, time series theory data science assumes random walk, however I assume variables are at least partly moved by free will of players of the game under some a priory unknown to me constraints and goals.

Should statistical solution advised work in such case? Can data predict reversal of possible current trend?

Yes, you can use scikit-learn for this use case. You may find this tutorial useful.


What is Reinforcement Learning? I am familiar with the concepts of Supervised and Unsupervised Learning but recently Reinforcement (Reinforced?) Learning also popped up before me a couple of times. Could anyone give a hint what is it, what is the main difference from the other two (or from the Unsupervised type, specifically), with some good example? Is it a promising alternative worth to explore or just some hyped niche curiosity?

Reinforcement Learning uses a simple logic of learning in which the network tries to learn from the feedback it obtains. This tries to optimise the overall reward in the long run instead of the current reward.

This is one of the best platform to read about it. It also contains some useful links.

As stated by the wiki,
The basic reinforcement learning model consists of:

a set of environment states S;
a set of actions A;
rules of transitioning between states;
rules that determine the scalar immediate reward of a transition; and
rules that describe what the agent observes.

The rules are often stochastic. The observation typically involves the scalar immediate reward associated with the last transition. In many works, the agent is also assumed to observe the current environmental state, in which case we talk about full observability, whereas in the opposing case we talk about partial observability. Sometimes the set of actions available to the agent is restricted (e.g., you cannot spend more money than what you possess).

Reinforcement learning(RL)  is the intersection of machine learning, decisions & control, and behavioral psychology. The intersection can be approached from all the three sides, and a detailed explanation is beyond the scope of a answer.
SO, I'll try to give a short account of all the three perspectives. Reinforcement Learning (RL) field, I give you here an illustrative simple example to understand the concept and also what is the relationship between Supervised Learning (SL) and Unsupervised Learning (UL).

Imagine that you are teaching your child to play super mario and you have two option  given the child the control stick and let him try to gain as many points as he can. or  you play one level at a time and ask him to do as you did so as to reach the end of the level like you did, that is you teach him how to play one level and ask him to play that or a similar level of the game.  This is SL as you will have for every input state of the road a mapping to the proper speed and points of jump and other attributes . The main point here is that you know what is the optimal thing and you try to teach your child to follow the similar steps. by doing so you teach him by examples.

In a RL setting, you just let the child try whatever it wants and you let the game give him a reward/punishment regarding the action(s) it takes.

In the SL example the child tries to minimize the error between your recommendation and its choices. In the RL example the child tries to maximize its reward by finding on its own what is the best to do. The SL approach at its best will lead you to a child to "mimics" what you taught it. In the RL approach at its best, the child will have a behavior that will be optimal in terms of playing Super Mario and also might be better than yours. In other words it will create its own strategy.

To sum up, in SL you have a teacher that tells you at every single time step exactly whats the correct response(think of math class where the teacher explains a identical problem to you on broad before giving you a problem to solve on you own). In RL you try and find it on your own and the teacher gives you a reward/punishment (think of a practical class where you are given a chemical and you have find its element/composition of the chemical and you follow you own intuition and do some trail and error to find the composition  of the chemical. In UL you don't have any external feedback. So uL falls between SL and RL(think of bio class where teacher gives you example about some classes of animals and ask you to do some classification. say, dog, cat, lion are vertebrate but snail, crab, earthworm are invertebrate so without knowing the meaning of invertebrate and vertebrate tell me which of these are invertebrate and vertebrate: frog, snake and grasshopper.

I simplified a lots just to give you a hint on the learning techniques with the example.


CNN tagging such that each input could have multiple tags Thanks in advance for reading my question!
I've been using CNNs to classify text using Keras and TF. My data is strings "I read the news" or "I read machine learning news" and my labels are tags: Data Science, Reporter, Child...
My issues is that each text can have multiple labels attached to it. How should I construct my target such that I can capture each of those multiple outputs?
Description, Tag
"I read the news", Child
"I read the news", Reporter
"I read machine learning news", Data Science
"I read machine learning news", Reporter

CNN's (generally) use softmax as activation function at the last layer, which gives a probability distribution over all possible labels. You use some kind of loss function to optimize the CNN to generate "right" label for a novel sample.

Loss function used in such case is cross-entropy loss or KL-divergence which measures how close two distributions are. For multiclass classification (different from your case, where each input example can have only one label), desired distribution is (0, 0, .... , 1, ..., 0) where 1 appears at the position of actual label. Loss function penalizes the distribution which is "far" from this distribution.

For your case you can encode your output as (0, 0, 1, 0, ... 1, ..,) so on where you put ones at positions of the labels which are associated with your input and use KL-divergence as loss function and optimize your CNN to  minimize the loss.


Usage of Precision Recall on an unbalanced dataset I am new to data science and am working on a dataset having roughly 213,000 rows and 31 columns. The 31st column being my Response variable having values 0's and 1's. Its a classification problem and the data set is unbalanced  as after I applied logistic regression to it, I got a model accuracy of 99.79%, however, by just counting the total number of 0's and 1's it would still show an accuracy of 99+% as it correctly classifies max no of 0's.

Also the confusion matrix is of no help too.

I did some digging up and learned to use precision recall in such scenarios. My question is should I even use logistic regression on this dataset? and then use precision recall?

If anyone can shed any light on what approach should I take, that would really help me move forward.

When evaluating your algorithms, especially when your dataset is unbalanced, you should use more metrics than just accuracy. The accuracy is how many examples you have correctly identified in total. As you have seen if you have an unbalanced dataset where 0.5% of your instances are 1's then this will result in 99.5% accuracy if you blindly set all your outputs as zeroes. This is obviously wrong albeit the high accuracy. The accuracy is calculated as

$Accuracy = \frac{\sum{TP} + \sum{TN}}{\sum{TP} + \sum{TN} + \sum{FP} + \sum{FN}}$

where TP is true positive, TN is true negatives, FP is false positives and FN is false negatives.

If you want to capture the performance of your unbalanced dataset you should look into the percentage of FP and FN you are calculating. You can do this using the sensitivity and the specificity. Calculate the sensitivity as

$Sensitivity = \frac{\sum{TP} }{\sum{TP}  + \sum{FN} }$

and the specificity as

$Specificity = \frac{\sum{TN} }{\sum{TN}  + \sum{FP} }$.

An ideal classifier should have the accuracy, specificity and sensitivity all be 1. This would mean every sample is correctly classified. In your case where you are getting very high false negatives, you will see that your sensitivity will be very low. This is a measure with which you can state that your algorithm is performing poorly. It is good form to always include these metrics in any statistical study you are doing. Accuracy alone is not sufficient to prove that you are obtaining good results.

Moreover, there is the receiver-operator curve (ROC). This will tell you your false positive rate for any true positive rate. You can then calculate the area under this curve (AUC) to get a comparable metric of performance.

All of these should be used together when exclaiming the performance of your algorithm. The ROC and AUC can be omitted however leaving out the sensitivity and specificity of your algorithm is unwise.

Found a solution, for an unbalanced dataset, first use SMOTE and then apply any model to use check AUC


Forecasting sales of next year using sales of past years? I have a dataset of sales of a company for different products throughout 3 years. I have to forecast the sales for each of these products for next year. A sample of the dataset is:

Here I have to forecast sales for year 4 from the sales record of each product over 3 years.

N.A is present where the particular product was not sold in that year.

Questions

1.Which algorithm should I use to forecast the sales for year 4? I have heard about ARIMA and xgboost being used for time series data. Can you please help me with that?

2.Now I am being given a new product which was not sold before. I am being asked to predict its sales for year 4, using the data for sales of other products:

What should be my approach for this problem?

Please help me out here. Thank you for your time and support in advance. I am sorry if my documentation or question is poor.

If you have daily data you could create a dummy time calendar, i.e. you create a dummy variable for each day of the week and include your company's promotions for each product, Christmas, Easter, public holidays.
Then use autoarima() or nnetar() (or combine them) to forecast the time horizon you want. This link is a good example.

Another possible way to do that is to use the method of following paper: "Hsiang-Fu Yu, Nikhil Rao, and Inderjit S. Dhillon. 2015. Temporal Regularized
Matrix Factorization. CoRR abs/1509.08333 (2015)" This paper can help you with with forecasting the new products too.

Or for forecasting new products you could find a comparable products from your historical data then use those comparables' sales data to forecast performance of the new ones.

If your data really looks like above I think you could use Hsiang-fu's paper. this is a good approach.

For autoregressive forecasting you could read from Rob Hyndman's blog.

To generate a forecast more accurately for the next year given past 3 years of history, you will need more granular data. At least monthly sales, so that the forecast can capture trend and/seasonality. With three data points, you will at best be able to generate a moving average forecast. Which isn't very good. If you have more granular data, use the forecast package in R. The forecast() function will pick the best method based on the time series pattern. You don't need to worry about stationarity or heteroscedasticity either.
For new products, you have two options. Explore diffusion curves such as Bass. Or, base the forecast curve on previous new product launches if there are shared attributes with existing products. If it's a brand new product line, evaluate market trends to generate the forecast. New product forecasting is a very difficult problem as such.


Once a predictive model is in production, how it can be evaluated? I have a data science project, predicting customer's next purchase day. Customer's one year behavioral data was split to 9 and 3 months for train and test, using RFM analysis, I trained a model with different classifiers and the best one's result is as follow:
Accuracy of XGB classifier on training set: 0.93
Accuracy of XGB classifier on test set: 0.68
This is my school's project, and I was wondering, in real world projects, how can we evaluate a model's performance after it's on production. How can I measure how successful my model was? What if the performance measures in production are much lower than my test result?

This is in fact a very good question.
The answer is simple, but depends on the case.
In general, what we do after pushing a model to production we apply an audit process. Let me explain: in reality machine learning models that are being pushed to production are pushed to replace another process (e.g, manual process- this is the case of automation). At the beginning everything predicted by machine learning model are audited through another process (e.g, manual), we call this stage the pilot stage. By comparing the model performance to the manual process we establish the quality of the model. Once we are happy, we start reducing the audit percentage from 100% to 5% or so ( there is some math behind what should be the audit percentage). This audit will never go away and will always be used to measure the performance of the model and to establish ground truth data for new samples that can be added to the training set.
In fact, training models in theory is something and using them in production is something else. It is really a complex process. Just to mention: we also like to implement some protection mechanisms to protect the model. For example data drift detection, uncertainty detection and so on.

After you train your model if you want to perform the predictions on new data
Something like this can help model.predict(new_data) . If your model performed good on the training data and very badly on the test data the main cause is overfitting(tThe model is overly accurate). Hope it helps.


Which Regression Machine Learning algorithm to use in Azure for statistical prediction/forecasting I've a golf website that pulls a load of statistical data and makes forecasts for upcoming events.

I'm trying to improve the algorithm and have been looking into Azure's ML Studio. I've lots of data about previous events and stats the competing golfers had and the time of the event so I'm confident I need to be using some sort of regression based algorithm but I'm not sure which one, there's a few that look appropriate?

Any advice would be greatly appreciated!
 Thanks

If I had to choose, it would be one of the last three as they are more sophisticated. If you have some time, try them all out, sweep the hyperparameters, and compare the results. This free EdX course has a segment where you do that comparison of different models in Azure ML:

https://www.edx.org/course/principles-machine-learning-microsoft-dat203-2x-4


Calculate future GDP % using machine learning I need to estimate the GDP % of a country three years into the future, based on historic data.

I have 30+ years of the following monthly data that includes features such as inflation and unemployment rate:

Year  Month  Inflation  Unemployment %  Other Features  GDP %
1990    1       1.1           6.2           ....         2.3
1990    2       1.3           6.1           ....         2.4

...
...
...

2019    6       0.8           4.8           ....         3.1
2019    7       0.9           4.9           ....         3.3

Using Random Forests (Python's scikit learn library), I can use the data to calculate the GDP of next month, 2019/8:

Once calculated (let's say GDP % = 3.2) I use this value and rerun the entire Random Forest process, including in the history data the value I obtained for 2019/8.

I continue iterating until I calculate 2022/8, and that is my result.

I have two issues:

It's a tedious process to recalculate everything on each iteration, if the historic data is large then this would take too long.
The 2022/8 result is obtained with just one path, I would need to rerun the overall process N times and take the average to have a more precise number.

I need this to work with machine learning (not necessarily Random Forests), any ideas how to improve/change this process?

I think you should consider time series modeling instead of observation based classification models. In the letter, you are propagating error in each prediction year.

I would use ARIMA, LSTMs, maybe semi-supervised models and motif discovery techniques.

I ended up using Recurrent Neural Networks, it's a good fit with time series.


Creating validation data for model comparison I am working on building a scoring algorithm for student data,

say the attributes are :

name, location, age, class, school_name, skill1, skill2, skill3

based on these data I need to create a student score.

I need to assign weight-ages for age, class, school_name skills and come up with a score for student.

say I have 2 scoring models like :

score_1 = x1*location_weight + x2*age_weight + x2*class_weight + x3*school_name_weight + x4*skill1_weight + x5*skill2_weight + x6*skill3_weight

score_2 = y1*location_weight + y2*age_weight + y2*class_weight + y3*school_name_weight + y4*skill1_weight + y5*skill2_weight + y6*skill3_weight

now how can I compare these models and evaluate them?

The problem is I don't have a test or validation set to prove or compare how accurate each of these model is, so in this case what is the best approach to compare and validate different models? also what is the best ways to build a validation set from scratch?

Predicting and scoring are two different tasks.

And according to your answers and comments you are not solving prediction problem. You just want to set to each student a number in range [1,100] according to some rule. This is ranking (or scoring, whatever).

Therefore, the terms #prediction_model, #accuracy, #validation, #training_set are out of this scope. You don't need to validate anything. You are not making predictions.

What you want is to map ranks to students.

But a problem is that you have mostely categorical data (school name, location etc) that cannot be 'ranked'. Some of them are useless at all: how does the student name refer to his school progress? :)

If you change it somehow to numerical (e.g. 'Skill_1_level', 'Skill_2_level', 'remoteness_of_location', 'school rank' etc) than you can do some ranking:

Normalize data: each of your factors

Multiple by 100, as you want [0,100] range instead of [0,1]
Set up weights based on your experience according to factor's importance. So that the sum of weights is 1.
And finally build a rank (score):

Rank = 0.1 * skill_1_level + 0.2 * skill_2_level + 0.05 * remoteness_of_location + 0.5 * school_rank + ...

I would start by fixing a seed (so that results are reproducible), then selecting a subsample (say, 10pct of data) and saving that as validation set. If you're using R, check the caret package

https://cran.r-project.org/web/packages/caret/index.html

and if Python is your weapon of choice, familiarize yourself with the sklearn documentation in 3.1:

http://scikit-learn.org/stable/modules/cross_validation.html

Both of those have multiple functions that might come in handy. If your computational resources allow it, I would suggest a cross-validation procedure (evaluating a model on multiple training-validation splits), both of the links I gave contain references to multiple functions that can help you achieve that.


About confidence/prediction intervals: parametric methods VS non-parametric (via bootstrap) methods About the methodology to find confidence and/or prediction intervals in, let's say, a regression problem, I know 2 main options:

Checking normality in the estimates/predictions distribution, and applying well known Gaussian alike methods to find those intervals if the distribution is gaussian
Applying non-parametric methodologies like bootstraping, so we do not need to assume/check/care whether our distribution is normal

With this in mind, I would basically always go for the second one because:

it is meant to be generic, as it does not assume any kind of distribution
it feels more like experimental as you can freely run as many iterations as you want (well, if it is computionally feasible)

The only drawback I could see is the computational cost, but it could be parallelized...

Can anyone give another point/advice?

The simplest type of bootstrapped confidence intervals, percentile intervals (described in simple terms here) are by no means the only way of creating bootstrapped CIs. You should think of bootstrapped confidence intervals as a family of techniques - there are some examples listed here along with some good information around the assumptions underlying the use of bootstrapped statistics.

How many original samples you need in order for bootstrapped confidence intervals to converge varies. Like the parametric approach, this convergence will arrive at the wrong value to the extent that your sample population is not representative of the global population you are estimating. A rule of thumb is to use parametric approaches if the sample population is n < 30, but in principle bootstrap could sometimes work with smaller samples (see this thread). If you have 30 samples or more, standard practice is to draw >1000 bootstrap samples, but the important thing is that you see the estimates converging.

If you want to use bootstrap CIs you should be really explicit about the approach you have used whenever you present your analysis, because different ways of producing them can lead to different intervals.


Are there any image classification algorithms which are not neural networks? Image classification is the task of assigning one of $n$ previously known labels to a given image. For example, you know that you will be given a couple of photos and each single image has exactly one of $\{cat, dog, car, stone\}$ in it. The algorithm should say what the photo shows.

The benchmark dataset for image classification is ImageNet; especiall thy large scale visual recognition challenge (LSVRC). It has exactly 1000 classes and a huge amount of training data (I think there is a down-sampled version with about 250px x 250px images, but many images seem to be from Flicker).

This challenge is typically solved with CNNs (or other neural networks).

Is there any paper which tries an approach which does not use neural networks in LSVRC?

To clarify the question: Of course, there are other classification algorithms like $k$ nearest neighbors or SVMs. However, I doubt they work at all for that many classes / that much data. At least for $k$-NNs I'm sure that prediction would be extremely slow; for SVMs I guess both fitting and prediction would be much to slow (?).

Part of the problem with answering this question is there are actually two questions.  The first:

Are there any image classification algorithms which are not neural networks?

Yes, lots. But now the actually question:

Is there any paper which tries an approach which does not use neural networks in LSVRC?

In your question, you rule out methods such as kNN and SVM because of speed. Bag of Words is one method used to solve this problem.  MATLAB has a good demonstration (http://www.mathworks.com/help/vision/examples/image-category-classification-using-bag-of-features.html).  But BoW incorporates k-means clustering, so that may not fit your needs.

There are some other interesting image classification methods such as texture analysis. TA is being researched as a way to classify malevolence of disease in medical images (such as tumors).  Here is a commonly referenced paper:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2701316/

Here's an overview of image classification:
http://www.tandfonline.com/doi/full/10.1080/01431160600746456

There are many algorithms that can be used to perform classifications ( many to the point that it is difficult to mention all of them )
I suggest you to have a look at this
http://dlib.net/ml_guide.svg

Making the decision which algorithm to use is a function of the problem you are working with, mainly:
1. The number of classes
2. The number of samples
3. The variations within classes and similarities between classes
4. Data imbalance
5. The dimension of your feature
And many other parameters

In general, CNN is very popular for two reasons:
They can lead to high performance in very challenging problems and they are general solutions in the context that you need to understand their architectures the strategies and tricks to perform training only, after that you do not need to change anything , no parameters to play with .


What does the long curve-shape t-SNE mean? I use 1-D CNN input 1*512 size time series data which randomly fragment segment, the output will classify input into 10 classes. After training the CNN, I apply t-SNE to the prediction which I fed in testing data.
In general, the output shape of the tsne result is spherical(for example,applied on MNIST dataset). But now I apply t-SNE on my own dataset. No matter how I adjust perplexity early, learning rate or maximum number of iterations. It will give me the result of long-shaped output, just like the pic below. Does the long-shaped t-SNE mean anything? Thanks everyone beforehand.

Edit: Explaining about dataset contents. p.s. before feeding into CNN for training, I randomly split it into training/testing dataset.

The lines therefore are the time series. t-sne is a very locally sensitive algorithm so every data point is very likely to show up nearest (in 2-D) to its nearest (N-D) neighbour.

In MNIST there is no sequence, so it doesn't show up as a line in 2-D. Every digit is written in a set of similar ways that cluster together. In your time-series data, if each time point is most similar to its next time point, what we expect is for them to be in some sequence - because you'd want point t=n to be closest to points t=n-1 and t=n+1  (if they are in fact the most similar).

This is also likely to be due to the fact that you're using features from a softmax (or other probability) layer. Make sure you are using a layer of activations and not probabilities. For example, in Keras, if you extract features from the last layer, it extracts the softmax probability vector. Probability vectors are very similar to each other, especially if they belong to the same class, since this is what softmax tries to do. I have seen similar shapes using time-series data when extracting probabilities and then clouds of data (instead of curves) when extracting previous layer activations.


How to detect time for the future events in time series data? I am dealing with IOT data from a mechanical machine. On the input I have ~100 features that are measured every minute. On the output, I have labels of zeros and ones, where zero indicates the absence of the event and 1 indicate the presence of an event. The event represent the failure for the machine in place. Therefore, the goal is to predict at every time step the remaining "minutes" for a failure to occur. I would like to know how to tackle this problem, and if possible for some material to read.
Is there a way to know which features in the past leads to a failure in the future if I'm using an LSTM?

Try boosting and random forests. You would need a long enough time series. A good reference (with examples in R) is
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2017). An Introduction to Statistical Learning: with Applications in R. Springer.
Usefulness of particular features can be assessed via variable importance plots.


Micro Average vs Macro average Performance in a Multiclass classification setting I am trying out a multiclass classification setting with 3 classes. The class distribution is skewed with most of the data falling in 1 of the 3 classes. (class labels being 1,2,3, with 67.28% of the data falling in class label 1, 11.99% data in class 2, and remaining in class 3)

I am training a multiclass classifier on this dataset and I am getting the following performance:

                    Precision           Recall           F1-Score
Micro Average       0.731               0.731            0.731
Macro Average       0.679               0.529            0.565

I am not sure why all Micro average performances are equal and also Macro average performances are low compared to Micro average.

Micro- and macro-averages (for whatever metric) will compute slightly different things, and thus their interpretation differs. A macro-average will compute the metric independently for each class and then take the average (hence treating all classes equally), whereas a micro-average will aggregate the contributions of all classes to compute the average metric. In a multi-class classification setup, micro-average is preferable if you suspect there might be class imbalance (i.e you may have many more examples of one class than of other classes).

To illustrate why, take for example precision $Pr=\frac{TP}{(TP+FP)}$. Let's imagine you have a One-vs-All (there is only one correct class output per example) multi-class classification system with four classes and the following numbers when tested:

Class A: 1 TP and 1 FP
Class B: 10 TP and 90 FP
Class C: 1 TP and 1 FP
Class D: 1 TP and 1 FP

You can see easily that $Pr_A = Pr_C = Pr_D = 0.5$, whereas $Pr_B=0.1$.

A macro-average will then compute: $Pr=\frac{0.5+0.1+0.5+0.5}{4}=0.4$
A micro-average will compute: $Pr=\frac{1+10+1+1}{2+100+2+2}=0.123$

These are quite different values for precision. Intuitively, in the macro-average the "good" precision (0.5) of classes A, C and D is contributing to maintain a "decent" overall precision (0.4). While this is technically true (across classes, the average precision is 0.4), it is a bit misleading, since a large number of examples are not properly classified. These examples predominantly correspond to class B, so they only contribute 1/4 towards the average in spite of constituting 94.3% of your test data. The micro-average will adequately capture this class imbalance, and bring the overall precision average down to 0.123 (more in line with the precision of the dominating class B (0.1)).

For computational reasons, it may sometimes be more convenient to compute class averages and then macro-average them. If class imbalance is known to be an issue, there are several ways around it. One is to report not only the macro-average, but also its standard deviation (for 3 or more classes). Another is to compute a weighted macro-average, in which each class contribution to the average is weighted by the relative number of examples available for it. In the above scenario, we obtain:

$Pr_{macro-mean}={0.25·0.5+0.25·0.1+0.25·0.5+0.25·0.5}=0.4$
$Pr_{macro-stdev}=0.173$

$Pr_{macro-weighted}={0.0189·0.5+0.943·0.1+0.0189·0.5+0.0189·0.5}={0.009+0.094+0.009+0.009}=0.123$

The large standard deviation (0.173) already tells us that the 0.4 average does not stem from a uniform precision among classes, but it might be just easier to compute the weighted macro-average, which in essence is another way of computing the micro-average.

I think the reason why macro average is lower than micro average is well explained by pythiest's answer (dominating class has better predictions and so the micro average increase).

But the fact that micro average is equal for Precision, Recall and F1 score is because micro averaging these metrics results in overall Accuracy (as micro avg considers all classes as positive). Note that if Precision and Recall are equal then F1 score is just equal to precision/recall.

As for the question if the "weighted macro-average" always going to equal the "micro average"? I did some experiments with different no. of classes and different class imbalance and it turns out that this is not necessary true.

These statements are made with assumption that we are considering all the classes of same dataset (in contrast to Rahul Reddy Vemireddy's answer)


Dealing with pre-trained model for grayscale images I would like to do Transfer Learning using one of the novel networks such as VGG, ResNet, Inception, etc.
The problem is that my images are grayscale (1 channel) since all the above mentioned models were trained on ImageNet dataset (which consists of RGB images).
One of the solutions is to repeat the image array 3 times to make it 3 channel.
Is this really the only solution for that? Is it a good solution? Are there any other solutions?

print(grayscale_batch.shape)  # (64, 224, 224)

rgb_batch = np.repeat(grayscale_batch[..., np.newaxis], 3, -1)
print(rgb_batch.shape)  # (64, 224, 224, 3)


Cosine similarity with arrays contaning NaN I am trying to calculate a cosine similarity using Python in order to find similar users basing on ratings they have given to movies. As it can be expected there are a lot of NaN values.
I am using movie dataset from Kaggle.

When I use np.dot() on two nd.arrays the outcome is:'nan'.
I have checked with np.nansum() that there are some other than 'nan' values.

I do not want to change all 'nan' values to '0' as it would mean that users have given 0 rating to the movies which lead to 'false' similarity between users.

Please, give me some advice regarding how to proceed with this problem.

Thanks in advance.

I think it's rarely meaningful to consider cosine similarity on sparse data like this, not just because of sparsity (because it's only defined for dense data), but because it's not obvious the cosine similarity is meaningful. For example a user that rates 10 movies all 5s has perfect similarity with a user that rates those 10 all as 1. Magnitude doesn't matter in cosine similarity, but it matters in your domain.

It's much more likely that it's meaningful on some dense embedding of users and items, such as what you get from ALS.

To answer the question, you either need to impute the missing ratings (don't assume 0, but a mean value or similar), or ignore dimensions that aren't defined in both.

def cosine_sim(df1, df2):

    df1na = df1.isna()
    df1clean = df1[~df1na]
    df2clean = df2[~df1na]

    df2na = df2clean.isna()
    df1clean = df1clean[~df2na]
    df2clean = df2clean[~df2na]

    # Compute cosine similarity
    distance = cosine(df1clean, df2clean)
    sim = 1 - distance

    return sim


How to make a decision tree with both continuous and categorical variables in the dataset? Let's say I have 3 categorical and 2 continuous attributes in a dataset. How do I build a decision tree using these 5 variables?
Edit:
For categorical variables, it is easy to say that we will split them just by {yes/no} and calculate the total gini gain, but my doubt tends to be primarily with the continuous attributes. Let's say I have values for a continuous attribute like {1,2,3,4,5}. What will be my split point choices? Will they be checked at every data point like {<1,>=1......& so on till} or will the splitting point will be something like the mean of column?

Decision trees can handle both categorical and numerical variables at the same time as features, there is not any problem in doing that.
Theory
Every split in a decision tree is based on a feature. If the feature is categorical, the split is done with the elements belonging to a particular class. If the feature is contiuous, the split is done with the elements higher than a threshold. At every split, the decision tree will take the best variable at that moment. This will be done according to an impurity measure with the splitted branches. And the fact that the variable used to do split is categorical or continuous is irrelevant (in fact, decision trees categorize contiuous variables by creating binary regions with the threshold).
Implementation
Although, at a theoretical level, is very natural for a decision tree to handle categorical variables, most of the implementations don't do it and only accept continuous variables:

This answer reflects on decision trees on scikit-learn not handling categorical variables. However, one of the scikit-learn developers argues that

At the moment it cannot. However RF tends to be very robust to categorical features abusively encoded as integer features in practice.

This other post comments about xgboost not handling categorical variables.
rpart in R can handle categories passed as factors, as explained in here
Lightgbm and catboost can handle categories. Catboost does an "on the fly" target encoding, while lightgbm needs you to encode the categorical variable using ordinal encoding.

Here's an example of how lightgbm handles categories:
import pandas as pd
from sklearn.datasets import load_iris
from lightgbm import LGBMRegressor
from category_encoders import OrdinalEncoder

X = load_iris()['data']
y = load_iris()['target']

X = OrdinalEncoder(cols=[3]).fit_transform(X)

dt = LGBMRegressor()
dt.fit(X, y, categorical_feature=[3])

When using Decision Trees, what the decision tree does is that for categorical attributes it uses the gini index, information gain etc.
But for continuous variable, it uses a probability distribution like the Gaussian Distribution or Multinomial Distribution to discriminate.
The second methodology is to convert it to categorical attributes and make rules like this: if a<100 and if a<100.


Anomaly detection with time series I have to study the behavoiur of a machine during its works: I have available time series of pressure, temperature and other physical measures for each work it is performing,
I would like to predict errors or failures but till now they have not been recorded yet. So I thought to using these time series to understand if there are abnormal behaviours (which could lead to a faillure), maybe clustering time series. Beacuse I suppose that the machine, in normal condition, has the same parameters for each work.
Is there a procedure/documentation/ best practises in order to do anomaly detecion with time series?
Thanks a lot!

You can extract features from timeseries sliding window, eg. mean, std, etc etc, and then use these as features into your model for anomaly detection

It's good u dont have any fault so far. You can build a normal model. may be relation of temprature with other variables. you can use regression or NN anything. Any deviation from established  relation may be an abnormality.-

this earlier post might give more insights-

Looking for good package for anomaly detection in time series


How sensitive are feedforward Neural Network to weight changes? Let's consider the space of feedforward neural networks with a given structure: $L$ layers, $m$ neurons per layer, ReLu activation, input dimension $d$, output dimension $k$.

Which means I'm considering the map $F: \mathcal{W}_1 \times \mathcal{W}_2 \times \dots \times \mathcal{W}_L \times \mathbb{R}^d \to \mathbb{R}^k$, where $\mathcal{W}_i$ is the space of possible weights for layer $i$. We also assume, for simplicity, that every weight matrix has a norm upper bounded by a constant $M$.
Let's now assume that I have fixed parameters so that we obtain $v = F(W_1, \dots, W_L, x^*) \in \mathbb{R}^k$ (note that $x^*$ is fixed as well).

Now imagine that I inject some random noise $\eta \in \mathbb{R}^{m \times m} $ in a weight matrix $W_i$, where the norm of the noise is 10% of the norm of the matrix, e.g. $||\eta|| = ||W_i||/10$  . How does it affect my final output?

which means, what's the expected value of $||v - v_*||$, where $v_*$ is the output of the network obtained after the small changes in the weights described before?

Note that this has nothing to do with the learning process, it's just about the sensitivity/resistance of a neural network with respect to random noise injected in a weight matrix.

you are asking for the condition I would say.
If you skim through the formulae, the german explanation is more
detailed.
In particular, the absolute condition at $x$ is defined as  $\kappa_{\text{abs}}:= \lim \sup _{\tilde{x} \rightarrow x} \frac{||f(x) - f(\tilde{x})||}{||x - \tilde{x}||}$, which means $\kappa_{\text{abs}} \geq 0 $ is the smallest number such that
there is a $\delta >0 $ so that all $\tilde{x}$ with $||\tilde{x}-x|| < \delta $ it holds that $|| f(\tilde{x})-f(x) || \leq \kappa_\text{abs} || \tilde{x} - x ||.$
The relative condition $\kappa_{\text{rel}} \geq 0 $ at $x$ is the smallest number such that there is a $\delta > 0 $ so that all $\tilde{x}$ with $||\tilde{x}-x|| < \delta $ satisfy:
$ \frac{|| f(\tilde x)-f(x) ||}{||f(x)||} \leq \kappa_\text{rel} \frac{|| \tilde{x} - x ||}{||x||}.$
The difference is thus, that the $\kappa_{\text{rel}}$ compares the relative change of the output with the relative change of the input.
In general, let $\tilde{f}$ denote an approximation of the real function $f$, $\tilde{x}$ the input with some noise, then there are 4 categories in numerics:

condition: Analyze $||f(x)-f(\tilde{x})||$
stability: Analyze $||\tilde{f}(x)-\tilde{f}(\tilde{x})||$
consistence: Analyze $||\tilde{f}(x)-f(x)||$
convergence: Analyze $||\tilde{f}(\tilde{x})-f(x)||$.

Your question is also related to adverserial attacks. You can have a look into the literature.
Note also that $\kappa_{\text{rel}} = \frac{||Df(x)||||x||}{||f(x)||}$, so that you could compute the condition number for any given neural network.

Its very hard to determine how exactly it would affect the learning of the network but from my experience, these are the possible scenarios. on how it can affect the output:

It might take your network a longer time to converge
Your network might never really reach the accuracy values as seen before adding the noise.
Certain weights can be dropped during learning or made to 0 when using a regularizer.

I am not quite aware if the affect on the output can be really modeled by a mathematical equation when a certain type of noise is added, if that is what you are looking for.


Shape extracting on 2D geometric data Given a set of lines, is there a way to train to extract geometric shapes. For example, the picture on left has some blue lines (with red endpoints). How can I train to extract shapes like on the right side? The output must be different shapes with each having a set of points.

I searched for related research papers. I could not find any paper with this exact problem, though some of them came close to this with RNN.

I haven't attempted this, but I'm pretty sure it would work reasonably well.

Identify all points that are connected to more than two lines. We know these must be vertex points.
For for all the points that are attached to only one line, identify their nearest neighbor and draw a line between them (this would connect that hole on the right wall of your house). All points should now be either labeled as a vertex or touching only two lines.
For each point touching two lines, calculate the angle between the two lines.
Find the min and max of these angles.
Use these as prototypes to classify angles into two proximity clusters using agglomerative clustering. Alternatively, pick a fixed angle value as your threshold (maybe around 120 degrees) and use that as the decision boundary for this classification.
The angles that clustered with the min correspond to the missing vertices.
Infer the vertex-vertex connectivity graph through the non-vertex points.(i.e. find all paths between pairs of vertices that do not pass through a third vertex).
Draw the graph.
Find the cycle's in the graph and color each one separately.

Everything doesn't have to be a big fancy neural network :)


Monitoring machine learning models in production I am looking for tools that allow me to monitor machine learning models once they are gone to production. I would like to monitor:

Long term changes: changes of distribution in the features with respect to training time, that would suggest retraining the model.
Short term changes: bugs in the features (radical changes of distribution).
Changes in the performance of the model with respect to a given metric.

I have been looking over the Internet, but I don't see any in-depth analysis of any of the cases. Can you provide me with techniques, books, references or Software?

The changes in distribution with respect to training time are sometimes referred to as concept drift.
It seems to me that the amount of information available online about concept drift is not very large. You may start with its wikipedia page or some blog posts, like this and this.
In terms of research, you may want to take a look at the scientific production of João Gama, or at chapter 3 of his book.
Regarding software packages, a quick search reveals a couple of python libraries on github, like tornado and concept-drift.
Update: recently I came across ML-drifter, a python library that seems to match the requirements of the question for scikit-learn models.

Something like this:

All credits goes to Abishek in any case as long as you save your desired differences, metrics, changes locally in your source code, you can hook it up with slack and receive messages. So in your cases for all 3 things its totally doable, you just have to get your hands dirty a bit (looks like a great hobby project!)


k-means and perceptron generalization ability I've implemented simple Widrow-Hopf perceptron and k-means clustering and compare results against MNIST data set. I didn't expect great results because of linear nature of these algorithms.  WH perceptron ended up with ~70% accuracy and k-means with ~50% (I have unit tests and compare with pure random chance 1/10 I suppose they do work somehow correct).

Also I gave "a hint" to k-means by set up initial centroids around different digits, so it can converge faster.

I suppose that k-means showed this result because digits are pretty much similar (6 is like 8 and 8 is like 9). Then I assumed that generalizations ability of WH perceptron is better than k-means. Did anyone see any article/book about this topic? I want to understand results from a rigor mathematical perspective.

Comparing K-means and perceptrons doesn't make sense, they are different types of algorithms.

K-means doesn't have a generalization ability at all, since it is an unsupervised algorithm, and generalization is a property of supervised learning algorithms.

If you are changing it to behave like a supervised algorithm, than it will have poor generalization ability compared to the perceptron, because K-means will have a tendency to overfit the data since it is forced to assign a cluster to each of its data points.

If you want to compare perceptron's performance to other algorithms, you should compare it to k-NN (K nearest neighbors), Random Forests, or Support Vector Machines.


Cluster elements that appear in the same lists Suppose I have a multitude of sets with (unordered) combinations of elements and I want to determine which elements tend to appear together.

For example

Given the following sets:

{a,e,g}
{a,e,h}
{a,e,i}
{b,f,j}
{b,f,k}
{b,f,l}
{d,c,m}
{d,c,n}
{d,c,o}

Element-pairs where both elements tend to occur in the same sets will have lower distance:

# Low-distance pairs:
{a,e}, {b,f}, {d,c}

# Medium-distance pairs
{a,g}, {b,j}, {d,m}, ...

# High-distance pairs:
{g,h}, {j,n}, {b,f}, ...

Currently

I'm implementing DBSCAN with a custom distance metric. I use the following distance metric between two elements:

d(a,b) = 1 - numsets(a, b) / (numsets(a,!b) + numsets(b,!a))

Where d(a,b) denote the distance between elements a and b. While numsets denotes how many sets fulfill some conditions:

numsets(a, b) - the number of sets that contain a and b
numsets(a,!b) - the number of sets that contain a but not b

This solution should achieve the goal however it's not a pretty solution and I couldn't find this problem on SE. In terms of solving the problem, is there a more sensible distance metric? In terms of implementation, is there a nicer way to do this?

That is a data mining problem, specifically affinity analysis.
One common method to solve it is the Apriori algorithm.

Levenshtein distance (and its cousing Jaro, Hemming etc...)

Levenshtein distance for measuring the difference between two sequences between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word (your case one set of characters) into the other.

There are a couple implementations, for example here, its the "edit_distance" function.


Primer on Random Forest Algorithm I wanted to ensure that I understood the random forest algorithm right. I think it's fundamental to understand how something work before using it!

Am I right with this:

If not: what did I understand wrong?

Ok. I think I got it now. It should be like this:

Am I right? If not: whats wrong?


anomaly detection - conceptual help on machine learning I am working on Anomaly detection model problem for a finance data set -  set of card transactions. My team member suggested an idea that
" First train the model with normal instances of data and to the trained model pass the anomalous data during testing to see whether the model will be able to detect the anomaly or not."
Is it possible like this with any Machine Learning algorithms. If so, can anybody suggest what algorithms can be used for this idea?

An additional question: How about the situation when there is no properly labelled fraudulent data and we have just transactional information. Does semi-supervised learning works? I do not have experience in this type of ML approach. Can anybody suggest good resources for this and also suggest can Semi-supervised learning or hybrid supervised learning works for anomaly/ fraud detection?

There is two options:

You have a label

Then what you probabibly have is a binary classification problem, probably unbalanced. If this happens follow this https://towardsdatascience.com/practical-tips-for-class-imbalance-in-binary-classification-6ee29bcdb8a7

You have no label

Then you have an unsupervised learning problem, in finance Isolation Forest is really used. If this is the case have a look at this https://towardsdatascience.com/outlier-detection-with-isolation-forest-3d190448d45e

How about Multivariate Gaussian?

Multivariate Gaussian finds means of all features, as well as covariance matrix of all features (you can use the normal instances for that). It uses this information to identify "how normal" a data point is. If the score is below a given threshold, the data point is an anomaly. You can supply it with known normal data points and known anomalies to test how it works.

*Normal data points that you use to test should not be part of normal data data points that you used to find means and covariance matrix.