Alternative distance to Dynamic Time Warping I am performing a comparison among time series by using Dynamic Time Warping (DTW). However, it is not a real distance, but a distance-like quantity, since it doesn't assure the triangle inequality to hold.

Reminder:d:MxM->R is a distance if for all x,y in M:

1 - d(x,y) ≥ 0, and d(x,y) = 0 if and only if x = y
2 - It is symmetric: d(x,y) = d(y,x)
3 - Triangle inequality: d(x,z) ≤ d(x,y) + d(y,z)

There is any equivalent measure that ensures the condition of distance in a matemathical sense? Obviously, I am not looking for a Euclidean distance, but one that ensures the proper classification of my series in a future clustering.
If so, there is any solid implementation in a R or Python package?

Like suggested in one answer on this SO question, you could use elastic matching with Levenshtein distance to your task. Levenshtein distance obeys triangle inequality and is therefore a metric distance.

Use of elastic matching was suggested for time series data comparison. Levenshtein distance works with characters data.

There is an implementation of elastic matching and Levenshtein distance calculation in Python.

To put them together you most probably need to build your own implementation.

When you say "However, it is not a real distance, but a distance-like quantity," you really mean, it is a measure, not a metric.

Why do you think you need a metric?

Consider the following common American girl names:

[Lisabeth, Beth, Lisa, Maryanne, Anne, Mary]

If asked to cluster these names into two groups, we would surely expect [{Lisabeth, Lisa, Beth}, {Maryanne, Mary, Anne}].

However, no distance measure that insists on the triangular inequality would give us “Beth” and “Lisa” in the same group; since they do not share a single character with each other. Yet, both share one character with “Anne”.

There is a tutorial on DTW here http://www.cs.unm.edu/~mueen/DTW.pdf


How to Make Meaningful Conclusions here? I recently appeared for an Interview for my college and I was asked the following question. The Interviewer said that this question was a Data Science question.
The question-
Suppose 7.5% of the population has a certain Bone Disease. During COVID pandemic you go to a hospital and see the records. 25% of the COVID Infected patients also had the Bone Disease. Can we say for sure if the Bone Disease is a symptom of COVID-19?
My Reponse-
I said No, and explained it as it's not necessary that COVID-19 is causing these symptoms, it could very well be possible that the 7.5% of the country's population which already had the disease is more susceptible to the virus due to lowered immunity. Hence making conclusions is not possible.
Then the interviewer asked me How can we be sure if it is a symptom or not?
I replied saying we can go to more Hospitals, collect more data and see if it correlates everywhere.
The Interviewer then said If we have the same results everywhere will you conclude it's a symptom?
I had no good answer but I replied that Just correlation of data is not sufficient, we also need to check if the people who have COVID-19 had the bone disease prior to getting infected or not. See if that percentage also correlates and stuff.
Here he stopped questioning however I couldn't judge If I was right or wrong.
I am in Grade-12 so I have no experience in Data Science as such. I do know a fair bit of statistics however I have never solved such questions. Can someone provide me insights on how to solve such questions and make meaningful conclusions?

It is very difficult (arguably impossible, if you want to get philosophical about it) to be absolutely, 100%, for sure about anything. For this reason, we talk in terms of probability/significance/confidence sets. A refresher on statistical hypothesis testing might help.
So to answer this type of question, people would usually try to attain a well-agreed-upon p-value for their problem, below which we can reject the null hypothesis, and above which we accept the null hypothesis. The null hypothesis in this case is that the bone disease is not a symptom of COVID, and the alternative would be that it is a symptom.

Edit for demonstration as requested in the comments:
Firstly, these methods are purely for association analysis, not for proving whether bone disease is a symptom of COVID—again, correlation $\ne$ causation!
Given that we are dealing with binary variables, you could use the Phi coefficient to measure the association of bone disease with COVID.
consider this contingency matrix:
|----------|---------|---------|-----------|
|          | Bone =0 | Bone =1 | total     |
| COVID =0 |    A    |  B      | I=A+B     |
| COVID =1 |    C    |  D      | J=C+D     |
|------------------------------|-----------|
|  total   |  K=A+C  |  L=B+D  | E=I+J+K+L |
|----------|---------|---------|-----------|

Which we can represent visually via a Venn diagram:

then you can calculate
$ \phi=\frac{AD-BC}{\sqrt{IJKL}}=\frac{ED-IK}{\sqrt{IK(E-I)(E-K)}} $
This is related to the Chi-squared test: $ \phi= \sqrt{\frac{\chi^2}{n}} $ So you can easily retrieve the p-value given that you know the degrees of freedom (in this case, it is just 1).
And you interpret it similar to the Pearson correlation coefficient (both from the same statistician—Pearson).

So given $7.5\%$ of the total has the bone disease, and $25\%$ of COVID patients have it, we can construct our contingency table in terms of $N$ (the number of samples). Where $Q$ is the percent of people with COVID:
$ I=N(1-Q)= $ number of people without COVID
$ J=NQ= $ number of people with COVID
$ K=N(1-0.075)=$ number of people without bone disease
$ L=N(0.075)=$ number of people with bone disease
We know that 25% of people with COVID also have the bone disease, so $D=J(0.25)\Rightarrow$
$\phi=\frac{E(0.25\cdot J)-IK}{\sqrt{IK(E-I)(E-K)}}$
Finally, we can calculate:
$\phi=\frac{0.25\cdot EQ-N(1-Q)(1-0.075)}{\sqrt{(1-Q)(1-0.075)(E-N(1-Q))(E-N(1-0.075))}}$
From here, we can find the associated p-value easily by looking it up in a Chi-Square p-value table, such as this one: http://chisquaretable.net/. Then you can accept/reject the null hypothesis given your predefined $\alpha$ threshold.

This is a valid question for hypothesis testing. However we only have partial data needed for the hypethesis test, so we can do one of two things:

Make a rough plausible estimate of the missing data (see Fermi problem)
Treat the missing data as parameters of the problem so we provide a parametrised answer.

The null hypothesis is that Bone disease and COVID19 are independent.
The alternative hypothesis is that they are not independent.
First approach

Percent of USA COVID-19 patients over total population can be estimated to be around $9\%$.
Percent of patients with Bone disease is given to be $7.5\%$.
Percent of patients having both COVID19 and Bone disease is given to be $25\%$.

We can assume, for simplicity and maximal unbiasedness, that the joint probability is a uniform random variable on $[0,1]$ interval.
If the null hypothesis is valid, then COVID19 and Bone disease are independent and the joint probability of a patient having both is the product of probability having one and the other. In other words:
$$P(COVID \& BONE) = P(COVID) \cdot P(BONE) = 9\% \cdot 7.5\% = 0.675\%$$
So by chance alone a patient has only $0.675\%$ chance to have both diseases.
Given percent of patients with both diseases is estimated as $25\% \cdot 9\% = 2.25\%$
Since we get $2.25\%$ we can reject the null hypothesis and conclude (with some significance) that COVID19 and Bone disease are not independent.
Second approach
Same as first approach but now take COVID19 percent as parameter and make parametrised conclusion.
Finally an even faster approach which goes directly to the point.
If the null hypothesis holds then the $7.5\%$ of patients with Bone disease over the population should remain same over the population of COVID19 patients (it is simply a subset of population and assuming plausible unbiased sampling, the same percent would prevail). In other words, we have no reason to believe the sample of COVID19 patients is not a representative sample of the population (this is a reflection of the fact that anybody can catch the virus the same, which is corroborrated by both virology and historical/statistical data on epidemics). So the percent of COVID19 patients having Bone disease should still be $7.5\%$. Since we get $25\%$ (it is with some significance that) we can reject the null hypothesis and the diseases are not independent.
Note the above of course do not constitute a complete statistical study, but only provided as example outline of a systematic process to follow (that can mark the beginning of a complete statistical study).


Python: Handling imbalance Classes in python Machine Learning I have a dataset for which I am trying to predict target variables.

Col1    Col2    Col3    Col4    Col5
  1      2       23      11     1
  2      22      12      14     1
  22     11      43      38     3
  14     22      25      19     3
  12     42      11      14     1
  22     11      43      38     2
  1      2       23      11     4
  2      22      12      14     2
  22     11      43      38     3

I have provided a sample data, but mine has thousands of records distributed in a similar way. Here, Col1, Col2, Col3, Col4 are my features and Col5 is target variable. Hence prediction should be 1,2,3 or 4 as these are my values for target variable. I have tried using algorithms such as random forest, decision tree etc. for predictions.

Here if you see, values 1,2 and 3 are occurring more times as compared to 4. Hence while predicting, my model is more biased towards 1 2 and 3 whereas I am getting only less number of predictions for 4 (Got only 1 predicted for policy4 out of thousands of records when I saw the confusion matrix).

In order to make my model generalize, I removed equal percentage of data that belongs to 1,2 and 3 value randomly. I grouped by each value in Col5 and then removed certain percentage, so that I brought down the number of records. Now I could see certain increase in percentage of accuracy and also reasonable increase in predictions for value 4 in confusion matrix.

Is this the right approach to deal with (removing the data randomly from those groups on which the model is biased)?

I tried for in-built python algorithms like Adaboost, GradientBoost techniques using sklearn. I read these algorithms are for handling imbalance class. But I couldnt succeed in improving my accuracy, rather by randomly removing the data, where I could see some improvements.

Is this reduction is undersampling technique and is this the right approach for under-sampling?

Is there are any pre-defined packages in sklearn or any logic which I can implement in python to get this done, if my random removal is wrong?

Also, I learnt about SMOTE technique, which deals with oversampling. Should I try this for value 4? And can we do this using any in-built packages in python? It would be great if someone helps me in this situation.

This paper suggests using ranking (I wrote it). Instead of using, for instance, SVM directly, you would use RankSVM. Since rankers compare observation against observation, training is necessarily balanced. There are two "buts" however: training is much slower, and, in the end, what these models do is rank your observations from how likely they are to belong to one class to how likely they are to belong to another so you need to apply a threshold afterwards.

If you are going to use pre-processing to fix your imbalance I would suggest you look into MetaCost. This algorithm involves building a bagging of models and then changing the class priors to make them balanced based on the hard to predict cases. It is very elegant. The cool thing about methods like SMOTE is that by fabricating new observations, you might making small datasets more robust.

Anyhow, even though I wrote some things on class imbalance, I am still skeptic that it is an important problem in the real world. I would think it is very uncommon that you have imbalance priors in your training set, but balanced priors in your real world data. Do you? What usually happens is that type I errors are different than type II errors and I would bet most people would be better off using a cost matrix, which most training methods accept or you can apply it by pre-processing using MetaCost or SMOTE. I think many times "fixing imbalance" is short to "I do not want to bother thinking about the relative trade-off between type I and II errors."

Addendum:

  I tried for in-built python algorithms like Adaboost, GradientBoost
  techniques using sklearn. I read these algorithms are for handling
  imbalance class.

AdaBoost gives better results for class imbalance when you initialize the weight distribution with imbalance in mind. I can dig the thesis where I read this if you want.

Anyhow, of course, those methods won't give good accuracies. Do you have class imbalance in both your training and your validation dataset? You should use metrics such as F1 score, or pass a cost matrix to the accuracy function. "Fixing" class imbalance is when your priors are different in your training and your validation cases.

When dealing with class imbalance problem you should mainly concentrate on error metric and you should choose F1 score as an error metric.
After choosing the correct metric we can use different Techniques for dealing with this issue.
If interested you can look into this blog, it is explained very nicely about the techniques used to solve this class imbalance problem.


Feature Importance - How to choose the number of best features? What is the standard or what method do you use to select part of features.

For ex: Using random forest, I got the following feature importances:

a : 25.4884726
b : 17.2736393
c : 12.3493490
d : 8.9383737
e : 8.1083837
f : 6.8272717
g : 4.1999999
...
...
...

For final prediction, you need to select only few features from the above. How do you decide the following:

  No. of features that you choose?
  Does the score matter when you choose?

This is an important problem, since many feature selection methods return feature scores/importances rather than a finite feature set. I currently know three approaches:

choose the k best attributes (fixed number defined)
choose the best k * 100% of attributes (relative number of features defined)
make a cutoff at the biggest difference in feature scores: all features are ordered according to their score and a split is made at the largest difference between one score and the next lower (biggest loss in importance)

You find a nice implementation doing this in R with the FSelector package.

I think, there are some research works tackling exactly this problem and may suggest better approaches, but I hadn't had the time to go deeper into this.

No. of features that you choose?

  Its depends on the number of classes you have to predict, let's explain this with example: consider we have 5 classes(labels) in our dataset and we choose only one feature, so tree has only one parent node and two leaf node which will accommodate only two classes and hence the accuracy decrease sharply, similarly if we choose only two features then we can only accommodate three classes and so one, so you have to consider more number of features as of classes and there after you have to test it with accuracy till there is no change in accuracy by adding extra features in the model.

Does the score(Variable Importance) matter when you choose?

  Yes, the score matter when deciding the features that you choose, since its depends on the Variable Importance of a feature is computed as the average decrease in model accuracy on the out of bag samples when the values of the respective feature are randomly permuted, so if you choose only the lower score variables for features then the accuracy sharply decreases.


Reducing Regression to Classification If a regression problem is reduced to classification, does minimizing the classification loss translate to minimizing regression error and hence better regression performance?

No, minimizing the classification loss does not translate to minimizing regression error and does not yield better regression performance.

This is a core tenant of Measure Theory. Regression measures outcome variables on ratio, interval, or ordinal scales. Classification measures outcome variables on nominal scales. You can always go from more complex scales (i.e., ratio, interval, or ordinal) to simpler scales (i.e., nominal). There are no guarantees about going back. This is technically called a non-injective mapping.

One way to think of it is that you are picking a downsampling scheme to map the data from regression to classification, and there are many possible mappings back from classification to regression. There are no guarantees on which mapping was used. Once downsampled, the data has to stay in the lower sampling space.

Well, that's my understanding of it.  To evaluate ML performance metrics, consider the following.

For classification, you can use.

Classification Accuracy
Log Loss
Area Under ROC Curve
Confusion Matrix
Classification Report

For regression, you can use.

Mean Absolute Error
Mean Squared Error
R^2

See the link below for all details.

https://machinelearningmastery.com/metrics-evaluate-machine-learning-algorithms-python/


Is it possible to get worse model after optimization? I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong.
in order to optimize my model I define param grid and than fit with the train data and then according to the results run again with nre parameters, e.g-
#ROUND 1
param_grid={
    'max_depth': [3,4,5],
    'learning_rate':[0.1,0.01,0.05],
    'gamma': [0,0.25,1.0],
    'reg_lambda':[0,1.0,10.0],
    'scale_pos_weight':[1,3,5]
}

grid_search = GridSearchCV(estimator = clf_xgb, param_grid = param_grid,
                          cv = 3, n_jobs = -1, verbose = 2)
grid_search.fit(X_train,y_train)
grid_search.best_params_

>>>.....

(and now based on the result changing the params...)
after this step I choose the best hyperparameters and run the model;
clf_xgb=xgb.XGBClassifier(seed=42,
                         objective='binary:logistic',
                         gamma=0,
                         learn_rate=0.7,
                         max_depth=6,
                         reg_lambda=0.8,
                         scale_pos_weight=1,
                         subsample=0.9,
                         cilsample_bytree=0.5)

clf_xgb.fit(X_train,
           y_train,
           verbose=True,
           early_stopping_rounds=10,
           eval_metric='aucpr',
           eval_set=[(X_test,y_test)])

The problem is that when I check the model score
clf_xgb.score(X_test,y_test)

I always get lower score than what I got before the optimization which makes me suspect that I'm missing something in the way doing it/basic principle in this process.
Is it possible that after running the optimization my score won't get better (and even worse?) ? Where is my mistake? Are there other parameters that could influence or improve my model?

Is it possible that after running the optimization my score won't get better (and even worse?) ?

Yes, theoretically, by pure luck, it is possible that your initial guess, before optimization of hyper-parameters, provides better results than the best of parameter combination found in the parameters grid. However, assuming you have enough data and your parameter grid is wide enough it is rather unlikely that the tuning of hyper-parameters would not be able to find better results. Such behavior rather indicates that something is wrong with your approach or your data.
If understand correctly, the choice of the best parameters is based on the cv results on training data, while in your final run the performance is assessed based on test dataset. If the distribution of training and test data differ significantly it could lead to the situation when the parameters providing the best results on the training data perform poorly on test data.

Where is my mistake?

As already mentioned by others, the parameters you are testing after the tuning were not included in the parameter grid. In this case it is incorrect to talk about the model performance "after running the optimization".
I suggest the following in order to investigate and fix the problem

Instead of using the hard-coded parameters in the XGBClassifier  call, use the optimal parameters found by tuning process, i.e. grid_search.best_params_. Furthermore, if you think that subsample and cilsample_bytree (a typo?) are relevant parameters include them in the parameters grid.
Increase the cv parameter to e.g. 5-10, the results with cv = 3 might be very unstable. You can assess the stability of your current results by using different random seeds and repeating the entire exercise.
Make sure that you use the consistent parameters in tuning process and in the final evaluation, or just include these parameters in the parameters grid if possible. In particular, check early_stopping_rounds and eval_metric.

Are there other parameters that could influence or improve my model?

From your code it is unclear how many rounds you use. Either increase n_estimators or include it in the parameters grid.
Given that you use AUCPR you might need to explicitly set the parameter maximize=True, otherwise in your final run you could minimize the AUCPR, which could explain poor results.

When you do hyper-parameter tuning, you improve the regularization of the model. Before you did optimization, you could be overfitting. After optimization you regularized your model and now it performs just right.

Then your model score will be worse after optimization on training set.
Your model score will also be bad for test set if your model heavily relies on one feature for classification.
You can use learning curve to see how the curve changes when you use optimization vs when you don't. And you can use df.corr() to see the correlation matrix for the correlation between feature values and target values.


In industry, what type of new data science algorithms does one develop? I've seen several job descriptions for data science which include developing a novel algorithm to be a part of production environments.
Can you give some input of what could be meant here exactly? Would they mean an algorithm that behaves somewhat of an ETL: getting the data, cleaning it, storing it, and running a known model on it? Or something more complex like building variations of known prediction algorithms? Some examples will really be nice, as I'm studying to go into this field.

I am no data scientist, only an aspiring one for two years, moving from my background in software engineering and mathematics. So I took some courses, had some interviews, read a lot on the subject online. My opinion:

New algorithms are developed in research centers and at universities. And even then, most algorithms used in companies are already developed, and just optimized even more.
Don't get your hopes up or be afraid that you have to reinvent gradient descent backpropagation.

With developing algorithms they most likely mean data extraction, data cleaning, data preparation for reporting statistics and presenting graphs. Maybe programmatically, maybe just using tooling.

The presented data may give more insights in simple relations in the domain, and maybe insights in more complex questions that can be asked.

You may get to define data flows, get to compare and select machine learning algorithms and tune its parameters. And continuously evaluate model performance in practice.

None.

What you do in business is program algorithms to solve biz problems.

Most all common problems have algorithms and processes already.  So what you would do is adopt and apply it to your particular situation.

Data science being the buzzword du jour, and very poorly understood by almost everyone as to what it is and how to use it, is more a fancy word to get you to apply your programming skills to databases in some fashion.

Often this will be some sort of data 'mining' that looks for patterns so that management can ASSume that correlation means causation and then use that as a rule to make decisions.

This is how the last wall street meltdown occurred.  They  made BAD ASSumptions that were close enough for most cases to ignore the errors in it but when that black swan hit they went bankrupt and the government had to bail them out with our tax money.


Implication of Central Limit theorem and its application in machine Learning According to central limit theorem, irrespective of the distribution of the original/population dataset, the sampling distribution will follow normal distribution.
My Question is if the means of n samples of the population data follow normal distribution, what is this helping us in ? I tried to find examples in the context of Ml but found no good example. It would be of great help if you could help me understand the significance of CLT ( basically application of knowing that sampling distribution will have bell curve) in the context of ML with an example.

Are you familiar with hypothesis testing? Confidence Intervals ? Statistical Significance ? All of these concepts typically rely on the central limit theorem.
One concrete example I can think of relating to ML: Regression
Most approaches to regression typically assume that the error term that appears follows a Gaussian distribution. How do we justify this assumption ? Well the idea is that we can think of the error term as a sum of multiple different error terms.
Now even if these individual error terms do not follow a Gaussian distribution, we can model their sum as a Gaussian because of the Central Limit Theorem.


Clustering with arrays / vectors as features? I am trying to cluster a large set of documents of which I have a DOC2VEC representation. But I want to cluster them with more features, thus resulting in having both a vector (numpy.asarray(doc2vec_for_document)) and single values as features.

Is this possible? I would like to try clustering with both K-means and DBSCAN of the SKLearn module in Python.

My dataset would look like this:
| Document | DOC2VEC | extra_ft1 | extra_ft2 | ...

Also, what if I had multiple vectors instead of 1 vector and many values?

You need to define a distance function that yields the desirable output.

Usually it will be enough if you can construct it such that d(a,b)for your purpose.

Doc2vec vectors are a bit tricky because they have so many dimensions, and a very strange geometry. It doesn't even seem to be clear whether cosine or Euclidean is to be used on these vectors...

Either way, you have to carefully balance the different features. In the other answer, minmax or stddev scaling was proposed. On one hand, that likely ruins the doc2vec properties, on the other hand this will but much more weight on the word vectors than on the other attributes.

For DBSCAN, you can also follow the "Generalized DBSCAN" approach. Here, the idea is just to define different thresholds for different features. Then neighbors must satisfy all thresholds. I.e. have doc2vec cosine less than A, and other features' distance less than B.
This is likely easier than balancing them as factors inside a single distance function.

But nothing saves you from weighting different feature (sets} carefully.

Yes, it is possible. First, you should concatenate the outputs of doc2vec with your extra feature vectors to form an augmented feature vector. Then, you must scale the augmented feature vector, using min-max scaling or standardization (zero mean, unit variance). That is because the different features should be in the same range to have a good behaviour in clustering task. You can find these pre-processing modules in scikit-learn:
http://scikit-learn.org/stable/modules/preprocessing.html


Parsing and storing a large amount of HTML data I have a data chunk (~30k) in which I have htmls pages and pngs saved in a folder for websites. These folders are titled based on some randomly generated hashes. My supervisor wants me to crunch through this data chunk and extract some attributes out of each HTML page and store it in a DB for future use. Attributes to be extracted comprises of page titles and copyright section from the HTML.
As per my understanding this data is unstructured because there is no relation per say in the folder data for now. Moreover, there is a somewhat inherent structure which is of HTML but essentially each page mutually disjoint with the rest which qualifies for unstructured. Please correct me if I am wrong here.

Manager wishes to have the data stored in an ELK stack. By storing, he is quite unclear at this point in time but so far he wants to have the whole HTML file, title and copyright for each single HTML file extracted and stored. Here comes my first concern which I need help with.

Is it a good idea to store whole HTML file into DB? I am of the
opinion that we place HTML files in a centralized storage on some
kind of FS and store the absolute paths of those files against each
entry in the DB (we are already doing the same thing for PNGs btw).

I haven't worked with ELK stack and I thought it would be a good learning opportunity. While going through online tutorials I have learned that it is essentially for logs parsing from different applications servers and storing and visualizing them in a presentable and searchable manner.

If anyone can comment about ELK, if it will work in my case, that
would be very helpful.

So far the end objective is to crunch through this data and store the attributes and when required search through the attributes and use them as per future need. For example, if there is a specific copy right text that is coming up very frequently, then get that copyright text and use it for classifying certain pattern which takes to my third and last question.

Will it help to store it in a non relational database and then query
accordingly? In my opinion RDBMS like mysql is a better contender
because it will be easy to search through the tables for a specific
type of title and then use it accordingly. End goal is not
visualization, but to have data at hand to use whenever required.

The terms "structured data" or "unstructured data" are not defined in such a way that a given dataset is always either one or the other. There are gray areas and I think this is one example. Since you cannot rely on the structure in your data, I would categorize this as unstructured.

To understand if it's a good idea to store the whole HTML in the DB (and same question for the PNGs), you need to weigh the pro's and con's. Pro storing everything in the DB is the simplicity: You don't have separate places where data is stored, so if you take a snapshot from the DB at some point in time and restore it, you restore the entire state as it was at that time. You do not need to worry about your disk storage separately, to restore that to a given state. Against storing everything in the DB is the amount of data. Can the DB handle it, or does performance suffer too much? Think about retrieving the data, searching/querying it, storing data, making back-ups. This will depend on your choice of database.

The same question for ELK and MySQL: What are the pros and cons of each? MySQL is simpler to install, that's always good. MySQL gives you a relational datamodel (tables can be related using foreign keys). Is that an advantage? MySQL gives you transactions. Is that helpful? ELK mainly gives you scalability, meaning that probably it would allow everything to be stored in the DB and still meet your performance needs.

If you can't store everything in MySQL (HTML and PNGs), then before choosing to store part of the data somewhere else, my first option would be to change DB technology to something that can store everything, rather than to start storing things in different places. So in that case ELK might be a good option, but store the PNGs there, too.


The cross-entropy error function in neural networks In the MNIST For ML Beginners they define cross-entropy as

$$H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i)$$

$y_i$ is the predicted probability value for class $i$ and $y_i'$ is the true probability for that class.

Question 1

Isn't it a problem that $y_i$ (in $\log(y_i)$) could be 0? This would mean that we have a really bad classifier, of course. But think of an error in our dataset, e.g. an "obvious" 1 labeled as 3. Would it simply crash? Does the model we chose (softmax activation at the end) basically never give the probability 0 for the correct class?

Question 2

I've learned that cross-entropy is defined as

$$H_{y'}(y) := - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$

What is correct? Do you have any textbook references for either version? How do those functions differ in their properties (as error functions for neural networks)?

One way to interpret cross-entropy is to see it as a (minus) log-likelihood for the data $y_i'$, under a model $y_i$.

Namely, suppose that you have some fixed model (a.k.a. "hypothesis"), which predicts for $n$ classes $\{1,2,\dots, n\}$ their hypothetical occurrence probabilities $y_1, y_2,\dots, y_n$. Suppose that you now observe (in reality) $k_1$ instances of class $1$, $k_2$ instances of class $2$, $k_n$ instances of class $n$, etc. According to your model the likelihood of this happening is:
$$
P[data|model] := y_1^{k_1}y_2^{k_2}\dots y_n^{k_n}.
$$
Taking the logarithm and changing the sign:
$$
-\log P[data|model] = -k_1\log y_1 -k_2\log y_2 - \dots -k_n\log y_n = -\sum_i k_i \log y_i
$$
If you now divide the right-hand sum by the number of observations $N = k_1+k_2+\dots+k_n$, and denote the empirical probabilities as $y_i'=k_i/N$, you'll get the cross-entropy:
$$
-\frac{1}{N} \log P[data|model] = -\frac{1}{N}\sum_i k_i \log y_i = -\sum_i y_i'\log y_i =: H(y', y)
$$

Furthermore, the log-likelihood of a dataset given a model can be interpreted as a measure of "encoding length" - the number of bits you expect to spend to encode this information if your encoding scheme would be based on your hypothesis.

This follows from the observation that an independent event with probability $y_i$ requires at least $-\log_2 y_i$ bits to encode it (assuming efficient coding), and consequently the expression
$$-\sum_i y_i'\log_2 y_i,$$
is literally the expected length of the encoding, where the encoding lengths for the events are computed using the "hypothesized" distribution, while the expectation is taken over the actual one.

Finally, instead of saying "measure of expected encoding length" I really like to use the informal term "measure of surprise". If you need a lot of bits to encode an expected event from a distribution, the distribution is "really surprising" for you.

With those intuitions in mind, the answers to your questions can be seen as follows:

Question 1. Yes. It is a problem whenever the corresponding $y_i'$ is nonzero at the same time. It corresponds to the situation where your model believes that some class has zero probability of occurrence, and yet the class pops up in reality. As a result, the "surprise" of your model is infinitely great: your model did not account for that event and now needs infinitely many bits to encode it. That is why you get infinity as your cross-entropy.

To avoid this problem you need to make sure that your model does not make rash assumptions about something being impossible while it can happen. In reality, people tend to use sigmoid or "softmax" functions as their hypothesis models, which are conservative enough to leave at least some chance for every option.

If you use some other hypothesis model, it is up to you to regularize (aka "smooth") it so that it would not hypothesize zeros where it should not.
Question 2. In this formula, one usually assumes $y_i'$ to be either $0$ or $1$, while $y_i$ is the model's probability hypothesis for the corresponding input. If you look closely, you will see that it is simply a $-\log P[data|model]$ for binary data, an equivalent of the second equation in this answer.

Hence, strictly speaking, although it is still a log-likelihood, this is not syntactically equivalent to cross-entropy. What some people mean when referring to such an expression as cross-entropy is that it is, in fact, a sum over binary cross-entropies for individual points in the dataset:
$$
\sum_i H(y_i', y_i),
$$
where $y_i'$ and $y_i$ have to be interpreted as the corresponding binary distributions $(y_i', 1-y_i')$ and $(y_i, 1-y_i)$.

Isn't it a problem that $y_i$ (in $\log(y_i)$) could be 0?

Yes it is, since $\log(0)$ is undefined, but this problem is avoided using $\log(y_i + \epsilon)$ in practice.

  What is correct?
  (a) $H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i)$ or
  (b) $H_{y'}(y) := - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log(1-y_i)})$?

(a) is correct for multi-class prediction (it is actually a double summation), (b) is the same as (a) for two-class prediction. Both are cross-entropy.

Example:

Suppose each training data $x_i$ has label $c_i' \in \{0, 1\}$, and model predicts $c_i \in [0, 1]$.

For 5 data points, true label $c_i'$ and model prediction $c_i$ are:

$(c_i', c_i)=\{(0, 0.1), (0, 0.4), (0, 0.8), (1, 0.8), (1, 0.2)\}$     (1),

Define vectors $y_i'$ and $y_i$ as

$y_{ik}':=1$ if $c_i'=k$, and $:=0$ otherwise,
$y_{ik}:=p(k|x_i)$ is the probability of $x_i$ belonging to class $k$, which is estimated by model.

Example (1) in $(y_i', y_i)$ notation turns into:

$(y_i', y_i)=\{([1, 0], [0.9, 0.1]),$ $([1, 0], [0.6, 0.4]),$ $([1, 0], [0.2, 0.8]),$ $([0, 1], [0.2, 0.8]),$ $([0, 1], [0.8, 0.2])\}$,

Both (a) and (b) are calculated as:

$H_{y'}(y)=-1/5([log(0.9)+log(0.6) + log(0.2)]_{c_i=0} + [log(0.8) + log(0.2)]_{c_i=1}) = 0.352$

Derivation:

Suppose there is multiple classes $1$ to $K$.
For training point $(x_i, c_i')$, $c_i' = k$ is equivalent to $y_i'=[0,..,1,0,..]$ which is 1 in $k^{th}$ position and 0 elsewhere. When $y_{ik}'=1$, we want model's output $y_{ik}=p(k|x_i)$ to be close to 1. Therefore, loss of $(x_i, k)$ can be defined as $-log(y_{ik})$, which gives $y_{ik} \rightarrow 1 \Rightarrow -log(y_{ik}) \rightarrow 0$. Loss over all classes can be combined as:

$L(y_i', y_i) = -\sum_{k=1}^{K}y_{ik}'log(y_{ik})$.

When $y_{ik}' = 1$, loss of all other classes $k' \neq k$ is disabled as $0log(y_{ik'})=0$, so for example when true label is $y_{im}'=1$, loss would be:

$L(y_i', y_i)=-log(y_{im})$.

Final formula over all training points is:

$H_{y'}(y)=-\sum_{(x_i, y_i')}\sum_{k=1}^{K}y_{ik}'log(y_{ik})$.

For binary classification, we have $y_{i0}' = 1 - y_{i1}'$ (true labels) and $y_{i0} = 1 - y_{i1}$ (model predictions), therefore (a) can be rewritten as:

$\begin{align*}
H_{y'}(y)&=-\sum_{(x_i, y_i')}y_{i1}'log(y_{i1})+y_{i0}'log(y_{i0})\\
&=-\sum_{(x_i, y_i')}y_{i1}'log(y_{i1})+(1-y_{i1}')log(1-y_{i1})
\end{align*}$

which is the same as (b).

Cross-entropy (a) over classes (one summation)

Cross-entropy (a) over classes is:

$H_{y'}(y)=-\sum_{k=1}^{K}y_{k}'log(y_{k})$,

This version cannot be used for the classification task. Lets reuse the data from the previous example:

$(c_i', c_i)=\{(0, 0.1), (0, 0.4), (0, 0.8), (1, 0.8), (1, 0.2)\}$

Empirical class probabilities are: $y'_0 = 3/5 = 0.6$, and $y'_1 = 0.4$,

Class probabilities estimated by model are: $y_0 = 3/5 = 0.6$, and $y_1 = 0.4$

(a) is calculated as: $-y'_0logy_0 - y'_1logy_1 = - 0.6log(0.6) -0.4log(0.4) = 0.292$.

Two data points $(0, 0.8)$ and $(1, 0.2)$ are miss-classified but $y'_0$ and $y'_1$ are estimated correctly!

If all 5 points where classified correctly as:
$(c_i', c_i)=\{(0, 0.1), (0, 0.4), (0, \color{blue}{0.2}), (1, 0.8), (1, \color{blue}{0.8})\}$ ,

(a) still remains the same, since $y'_0$ is again estimated as $y_0=3/5$.


What are deconvolutional layers? I recently read Fully Convolutional Networks for Semantic Segmentation by Jonathan Long, Evan Shelhamer, Trevor Darrell. I don't understand what "deconvolutional layers" do / how they work.

The relevant part is

  3.3. Upsampling is backwards strided convolution

  Another way to connect coarse outputs to dense pixels
  is interpolation. For instance, simple bilinear interpolation
  computes each output $y_{ij}$ from the nearest four inputs by a
  linear map that depends only on the relative positions of the
  input and output cells.
  In a sense, upsampling with factor $f$ is convolution with
  a fractional input stride of 1/f. So long as $f$ is integral, a
  natural way to upsample is therefore backwards convolution
  (sometimes called deconvolution) with an output stride of
  $f$. Such an operation is trivial to implement, since it simply
  reverses the forward and backward passes of convolution.
  Thus upsampling is performed in-network for end-to-end
  learning by backpropagation from the pixelwise loss.
  Note that the deconvolution filter in such a layer need not
  be fixed (e.g., to bilinear upsampling), but can be learned.
  A stack of deconvolution layers and activation functions can
  even learn a nonlinear upsampling.
  In our experiments, we find that in-network upsampling
  is fast and effective for learning dense prediction. Our best
  segmentation architecture uses these layers to learn to upsample
  for refined prediction in Section 4.2.

I don't think I really understood how convolutional layers are trained.

What I think I've understood is that convolutional layers with a kernel size $k$ learn filters of size $k \times k$. The output of a convolutional layer with kernel size $k$, stride $s \in \mathbb{N}$ and $n$ filters is of dimension $\frac{\text{Input dim}}{s^2} \cdot n$. However, I don't know how the learning of convolutional layers works. (I understand how simple MLPs learn with gradient descent, if that helps).

So if my understanding of convolutional layers is correct, I have no clue how this can be reversed.

Could anybody please help me to understand deconvolutional layers?

Deconvolution layer is a very unfortunate name and should rather be called a transposed convolutional layer.

Visually, for a transposed convolution with stride one and no padding, we just pad the original input (blue entries) with zeroes (white entries) (Figure 1).

In case of stride two and padding, the transposed convolution would look like this (Figure 2):

All credits for the great visualisations go to

Vincent Dumoulin, Francesco Visin - A guide to convolution arithmetic for deep learning

You can find more visualisations of convolutional arithmetics here.

The following paper discusses deconvolutional layers.Both from the architectural and training point of view.Deconvolutional networks


Encoding for k-level qualitatative variable I have a qualitative variable, e.g. userId, which could take around 30,000 different coded values ($k$). I would like to represent this variable as a dummy variable. Coding this into a vector of size $k$ doesn't seem to be a good approach. Is there a more compact method for coding for this variable?

There is always clustering techniques you could do that would help you determine how to group userId into a dummy variable.

If you are using tree based methods, you can replace the qualitative variables with random numbers or with the counts of each class.


How to determine the number of the training images in Keras after data augmentaion? I want to create a CNN model and I am using data augmentation. I want know the number of augmented images in Keras.

How to determine the number of the training images in Keras after data augmentation?

For example, if I have 2000 images for training and I am applying data augmentation, what is the number of the images after augmentation?

The images in the dataset are not used directly. Instead, only augmented images are provided to the model. So, your answer is 2000 images

You can generate as many as augmented images in Keras using ImageDataGenerator. I struggled with the same problem and figured out that there is an infinite number of images you can generate. Second thing is you can limit the number of images to be generated.


Would like to know some Machine Learning applications for Civil Engineering I've thought about some, like predicting Concrete's strength by it's composition, but would like some ideas on the theme. One other thing I was thinking was traffic, how would I use machine learning to predict something related to streets and roads... Maybe something related to hydraulics on a building. I'm really beginning on machine learning so I'm not so good at knowing how can I use it. Although it's amazing. Thank's a lot guys.

This is something I've experimented with substantially.  Predicting the compressive strength of concrete based on it's mix design is a great example, and fairly challenging - I used the concrete compressive strength dataset available at the UCI ML repository to help me better understand how to manipulate ML algorithms to get good predictive results.  Here's a guy who took the same data set and used BigML's tools to do an analysis.

I've also used ML to predict payitem bid prices on highway construction contracts to help improve cost estimation.  For example, I set out to determine the unit price for a specific item, like guardrail, that contractors are likely to bid given a contract's location, time of year, total value, relevant cost indices, etc.  It took a lot of time and experimentation, but I achieved surprisingly good predictive results.

Regarding your specific topics of interest, I imagine you'll have to just do a lot of searching and reviewing master's theses on related topics, taking time to learn the mathematical models and theories that they employ.

If you want to know where to get started, my learning path involved the following:

Learning whatever I could about neural networks and machine learning
theory (guys like Jeff Heaton proved helpful, here)
Taking Andrew Ng's Machine Learning course on Coursera.
Downloading and getting to know the R scripting language
Reading whitepapers / theses on applying ML to specific
applications.
Developing a solid, rigorous understanding of basic statistics theory, especially linear regression.

There's a lot of different ways to approach the topic, though.  Websites like BigML.com are a good way to get your feet wet doing ML without having a rigorous mathematical / statistical understanding, but it'll only get you so far.

In the end, you'll really need to devote time to getting your head wrapped around the math and programming in languages like R or Python.  It's also important to remember that while ML seems to favor complexity over simplicity (in many cases non-linear problems benefit from novel and sometimes non-intuitive approaches), sometimes the simplest approach is best.  It's really a question of how accurate the final model really needs to be.

Machine Learning is not an answer for the question of Life, the Universe, and Everything or silver bullet but a very powerful tool which you should apply to a problem (and corresponding data). So, the short answer to your question is 'depends on data you have and problems you see'.

Generally, the best way would be taking any available introductory course into Data Science / Machine Learning, given you have at least some programming experience so that you could complete assignments: that would not address your question directly, but you will get your hands on some examples of data and will see how ML is used to tackle that data. After you will grow some understanding, you will see potential applications of ML when you see any particular blob of data.


Merging multiple data frames row-wise in PySpark I have 10 data frames pyspark.sql.dataframe.DataFrame, obtained from randomSplit as (td1, td2, td3, td4, td5, td6, td7, td8, td9, td10) = td.randomSplit([.1, .1, .1, .1, .1, .1, .1, .1, .1, .1], seed = 100) Now I want to join 9 td's into a single data frame, how should I do that?

I have already tried with unionAll, but this function accepts only two arguments.

td1_2 = td1.unionAll(td2)
# this is working fine

td1_2_3 = td1.unionAll(td2, td3)
# error TypeError: unionAll() takes exactly 2 arguments (3 given)

Is there any way to combine more than two data frames row-wise?

The purpose of doing this is that I am doing 10-fold Cross Validation manually without using PySpark CrossValidator method, So taking 9 into training and 1 into test data and then I will repeat it for other combinations.

Stolen from: https://stackoverflow.com/questions/33743978/spark-union-of-multiple-rdds

Outside of chaining unions this is the only way to do it for DataFrames.

from functools import reduce  # For Python 3.x
from pyspark.sql import DataFrame

def unionAll(*dfs):
    return reduce(DataFrame.unionAll, dfs)

unionAll(td2, td3, td4, td5, td6, td7, td8, td9, td10)

What happens is that it takes all the objects that you passed as parameters and reduces them using unionAll (this reduce is from Python, not the Spark reduce although they work similarly) which eventually reduces it to one DataFrame.

If instead of DataFrames they are normal RDDs you can pass a list of them to the union function of your SparkContext

EDIT: For your purpose I propose a different method, since you would have to repeat this whole union 10 times for your different folds for crossvalidation, I would add labels for which fold a row belongs to and just filter your DataFrame for every fold based on the label

You could do it like this.

val myDFCsv = spark.read.format("csv")
   .option("sep","|")
   .option("inferSchema","true")
   .option("header","false")
   .load("mnt/rawdata/2019/01/01/client/ABC*.gz")

myDFCsv.show()
myDFCsv.head()
myDFCsv.count()

//////////////////////////////////////////
// If you also need to load the filename
val myDFCsv = spark.read.format("csv")
   .option("sep","|")
   .option("inferSchema","true")
   .option("header","false")
   .load("mnt/rawdata/2019/01/01/client/ABC*.gz")
   .withColumn("file_name",input_file_name())

myDFCsv.show(false)
myDFCsv.head()
myDFCsv.count()


Logistic regression in python I have done linear and multivariate regression so I understand what probability, cost and gradient descent functions are. I do not understand what the last 3 formulas mean and how they relate to each other. I don't expect anyone to explain them to me but could someone please share a link that explains them in simple terms? I've tried looking online but the formulas used often look different from the ones below.

Sorry for posting a picture like this but I don't know how to format formulas.

Logistic Regression is rather a hard algorithm to digest immediately as details often are abstracted away for the sake of simplicity for practitioners. To explain the idea behind logistic regression as a probabilistic model, we need to introduce the odds ratio, i.e. the odds in favor of a particular event. The odds ratio can be written $\frac{p}{1-p}$, where p stands for the probability of the positive event. We can then further define the logit function, which is simply the logarithm of the odds ratio (log-odds). The logit function takes input values in the range 0 to 1 and transforms them to values over the entire real number range, which we can use to express a linear relationship between feature values and the log-odds:

$logit \frac{p}{1-p} = w_{0}x_{0} + w_{1}x_{1}...w_{n}x_{n}= w^{T}x $ (1)

In Logistic Regression we are interested in predicting the probability that a certain sample belongs to a particular class, which is the inverse form of the logit function. It is also called the logistic function, sometimes simply abbreviated as sigmoid function:

$p(Y=1|x) = \phi(z=W^Tx) = \frac{1}{1+e^{-z}}$ (2)

$ p(Y=0|x) = 1 - p(Y=1|x) $ (3)

Here it comes the tricky part, the parameter estimations i.e. w (see eq. 1). It basically states that a solution to a Logistic Regression problem is the set of parameters w that maximizes the likelihood which is defined via maximum likelihood estimator (recall that we are going to define a loss function going from here). As logistic regression predicts probabilities, rather than just classes, we can fit it using likelihood:

$ L(w) = P(Y_{i}|w;x) = \prod_{i=1}^{n} \phi(z)^{Y_{i}}(1-\phi(z))^{1-Y_{i}}$ (4)

In practice, instead of maximizing this likelihood, it is easier to maximize the (natural) log of this equation, which is called the log-likelihood function:

$log(L(w)) = \sum\limits_{i=1}^n [Y_{i}log\phi(z)+(1-Y_{i})log(1-\phi(z))]$ (5)

Or to make it mathematically more convenient, we take the negative of this function, so that it can be minimized using gradient descent:

$ cost = - log(L(w)) = l_{i}(w) = \sum\limits_{i=1}^n [- Y_{i}log\phi(z)-(1-Y_{i})log(1-\phi(z))]$ (6)

The cost would look like this (taken from here):

Here then we take derivative with respect to parameters w (see eq. 1), rather than z (in your photo it is with respect to $\alpha$). Those derivations would be like what you showed. Still I recommend checking out this blogspot that goes through these steps in more details, or parts of implementation in python here, if needed.

I will suggest, take the start from here. enter link description here


Augmenting the Cost Function with a Gaussian Prior I'm doing an online course and one of the assignment says:

  To avoid parameters from exploding or becoming highly correlated, it is helpful to augment
  our cost function with a Gaussian prior: this tends to push parameter weights closer to zero, without
  constraining their direction, and often leads to classifiers with better generalization ability.

Exactly what does "augment our cost function with a Gaussian prior" mean, and how could I go about doing that?

I tried searching on google but didn't find anything helpful.

Let's look at the example of linear regression. Instead of deriving it from solving the normal equations, we can motivate it by thinking about it as finding the conditional distribution $P(y|x)$. Let's assume that this distribution follows a Gaussian distribution with fixed variance $\sigma^2$ and mean $\hat{y}(x,w)$ and weights $w$. Assuming that samples are i.i.d. we can easily show that applying maximum-likelihood gives the same values of $w$ than minimizing the mean squared error. See example 5.5.1 for a mathematical explanation.

If we additionally assume a prior Gaussian distribution for $P(w)$ (w.l.o.g. we can assume unit variance $I$), we can now show that the posterior $P(w | x,y)$  of $w$ is also Gaussian distributed (that is because the prior is "conjugate") with variance $\frac{1}{\alpha} I$ and we recover linear regression with weight decay $\alpha w^T w=\alpha \lVert w \rVert^2_2$  (i.e. $L^2$ or Tikhonov regularization).

Intuitively, a Gaussian distribution's probability mass is centered around the mean (assumed here to be zero). If we assume such a distribution for the coefficients in the linear regression (as our prior knowledge), linear regression will favor coefficients close to zero.

All of this is nicely explained here (chapter 5) or here (chapter 18).

It was hard to look for anything based on the information. Still, I found this interesting paper- Augmented Functional Time Series Representation and Forecasting with Gaussian Processes on the web.

Just explore it. Hope it will help you upto some extent. Cheers!


Do I need to buy a NVIDIA graphic card to run deep learning algorithm? I am new in deep learning. I am running a MacBook Pro yosemite (upgraded from Snowleopard). I don't have a CUDA-enabled card GPU, and running the code on the CPU is extremely slow. I heard that I can buy some instances on AWS, but it seems that they don't support macOS.

My question is, to continue with the deep learning, do I need to purchase a graphic card? Or is there other solution? I don't want to spend too much on this...

I would recommend familiarizing yourself with AWS spot instances. It's the most practical solution I can think of for your problem, and it works your computer too. So, no you don't have to buy an Nvidia card, but as of today you will want to use one since almost all the solutions rely on them.

AWS does "support" Mac OS. You can use any SSH client from your Mac to access the GPU instance.


Clustering sequences of sentence embeddings I have a sequence of events, right now I am not worried about their actual times, just the order. This is a sequence of web page views.

I have modelled my data as the following, where each element represents the category of the web page.

user_sequence = ['A', 'A', 'B', 'C', ...]

Following this I used the code and approach from this paper: Sequence Graph Transform

My question is how could I represent more complex data in my sequences, for example, we have an embedding representing the page content, along with features including the dwell of the user on each page.

So to summarise, the goal is to do process sequences of rich event data in an unsupervised manner.

The approach taken for anyone coming across this has been to first cluster each web page / node before passing it into SGT. This means we can encode more information into the sequence prior to using SGT.


As with anything, it depends. :) Your question will very much depend on the model.

This question is reminiscent of the broader debate about discriminative versus generative models. Usually, we work with discriminative models. That is, you model $\text{P}(Y=1\,|\,X_1, X_2, \dots)$, where $Y$ are your classes and $X_i$ your observations.

There is another class of models: generative models, like Restricted Boltzmann Machines, which model joint distributions, $\text{P}(Y,X_1, X_2, \dots)$. They can be used for classification because you can easily turn a joint probability into a conditional probability. These models usually are broader in scope, and usually do not perform as well, which is why they aren't used as much. But they do exist.

Assuming that you are talking about discriminative models, like decision trees or the logistic regression, then what you want is how to invert this conditional probability, so that you have a distribution along $X_i$; that is, $\text{P}(X_1=x_1, X_2=x_2, \dots \,|\, Y)$.

From Bayes's theorem, $\text{P}(Y=1\,|\,X_1=x_1, X_2=x_2, \dots)=\text{P}(X_1=x_1, X_2=x_2, \dots \,|\, Y=1)\frac{\text{P}(Y=1)}{\text{P}(X_1=x_1, X_2=x_2, \dots)}.$

So, yes, you can. Where is the catch?

$X$ and $Y$ have to be discrete. It is very easy to create a probability density function from categorical variables by just getting conditional probabilities out of your model by just trying all combinations of $X$. And, once you have a probability density function, then you can just use a sampling technique to generate values.

It is less clear to me how you could make this work for continuous values, unless you are willing to use an approximation by discretizing your observations. I think an exact solution will depend very much on the discriminative model at hand. You cannot do this alone by getting conditional probabilities out of the model.

But if sampling is important for you, then you should just use a generative model.

Let s1 to s4 be the signs (+1 or -1) of the regression coefficients of f1 to f4.

Then the predicted value for a data item with columns s1*Inf to s4*Inf will be (as near as precision allows it) 1. You go to the extreme of the parameter space in the right direction (as told by the coefficients signs).

Inverting the signs and doing the same should give you a data item that is predicted to be 0.

Example, code in R

Create sample data set.

set.seed(12345)
d2 = data.frame(f1=runif(10), f2 = runif(10), f3 = runif(10),
      output=sample(c(0,1),10,TRUE))

Fit model:

m2 = glm(output~f1+f2+f3, data=d2, family="binomial")

Look at model parameters:

coef(m2)
## (Intercept)          f1          f2          f3
##   5.258817   -4.848013  -15.370838    4.890984

This means as f1 and f2 increase, the fitted probability of zero increases because their signs are negative. As f3 increases, the fitted probability of one increases because it has a positive sign. So....

predict(m2, newdata=data.frame(f1=-Inf, f2=-Inf, f3=Inf), type="response")

should be one, which it is, and:

predict(m2, newdata=data.frame(f1=Inf, f2=Inf, f3=-Inf), type="response")

should be zero, which it is, within tolerance.

Zeroes and ones in logistic regression responses are like plus or minus infinity in a linear regression, so if you think back to a simple X-Y fit, you get an infinite response to a linear regression when you go to +Inf on the X-axis if the slope is positive or -Inf on the X-axis if the slope is negative. Transform that to a logistic regression and then you get to +1 or 0 at the appropriate signed infinities of the covariates.


How to generate a output like this? can u please give an explanation for this ?

Using the example dataframe provided by@JahKnows:

import pandas as pd

df = pd.DataFrame(data={'shopify_customer_id': [1,1,2,3,4,5],
                        'financial_status': ['paid', 'refunded', 'paid', 'refunded','paid', 'paid'],
                        'count': [8, 1, 13, 1, 1, 1],
                        'order_amt': [19, 19, 0, 19, 0, 24]})

You could also do the following:

df=df.groupby(['shopify_customer_id','financial_status']).sum().unstack()
df.columns = ['_'.join(col).strip() for col in df.columns.values]
df.reset_index(inplace=True)
df.fillna(0,inplace=True)

That gives you this (if this is what you intend to have!):

   shopify_customer_id  count_paid  count_refunded  order_amt_paid  order_amt_refunded
0                    1         8.0             1.0            19.0                19.0
1                    2        13.0             0.0             0.0                 0.0
2                    3         0.0             1.0             0.0                19.0
3                    4         1.0             0.0             0.0                 0.0
4                    5         1.0             0.0            24.0                 0.0

The dataframe can be defined as follows

import pandas as pd

df = pd.DataFrame(data={'shopify_customer_id': [1,1,2,3,4,5],
                        'financial_status': ['paid', 'refunded', 'paid', 'refunded', 'paid', 'paid'],
                        'count': [8, 1, 13, 1, 1, 1],
                        'order_amt': [19, 19, 0, 19, 0, 24]})

This gives the table you described above. Now we will perform the transformation. We will use a dictionary to keep track of all the customers in the database. The first if statement in the code below will add customers not yet in the list. Then we will check if the transaction is a 'sale' or a 'refund'. And we will update the dictionary for the given customer accordingly.

shopify_customers = {}
for i in df.iterrows():
    row = i[1]
    if row['shopify_customer_id'] not in shopify_customers:
        shopify_customers.update({row['shopify_customer_id']:
                                  {'shopify_customer_id': row['shopify_customer_id'],
                                   'order_count_paid': 0,
                                   'order_count_refunded': 0,
                                   'order_amt_paid': 0,
                                   'order_amt_refunded': 0}})

    if row['financial_status'] == 'paid':
        shopify_customers[row['shopify_customer_id']]['order_count_paid'] += row['count']
        shopify_customers[row['shopify_customer_id']]['order_amt_paid'] += row['order_amt']

    elif row['financial_status'] == 'refunded':
        shopify_customers[row['shopify_customer_id']]['order_count_refunded'] += row['count']
        shopify_customers[row['shopify_customer_id']]['order_amt_refunded'] += row['order_amt']

Finally we will put the data into a pandas DataFrame

pd.DataFrame(data = [shopify_customers[i] for i in shopify_customers])


Convert Neural network to Keras Classifier I am training a Neural Network for Multi-Class classification. After succesfully training it and validating the model through cross validation, I would like to use this network inside a voting Classifier. In order to perform cross validation on my trained network I convert it to a Keras Classifier and then calculate its validation score. However when I parse the same exact "Keras Classifier" in the Voting Classifier method, I get the following error:
ValueError: The estimator KerasClassifier should be a classifier

The code can be seen below:
import random
random.seed(42)

from keras.layers import Dense
from keras.models import Sequential
from keras.wrappers.scikit_learn import KerasClassifier
import tensorflow as tf
from sklearn.ensemble import VotingClassifier

def NeuralNetwork():
    model = Sequential()
    # define first hidden layer and visible layer
    model.add(Dense(600, input_dim=k, activation='relu'))
    model.add(Dense(20, activation='relu'))
    # define output layer
    model.add(Dense(3, activation='softmax'))
    # define loss and optimizer
    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=["accuracy"])
    return model

NN = KerasClassifier(build_fn=NeuralNetwork , epochs=100, batch_size=100, verbose=0)
cross_val_score(NN, X_new_train,y_train_no_id, cv=3)

votingC = VotingClassifier(estimators=[ ('LR1', LR1),('LR2', LR2), ('XGB',XGB),('NN',NN)], voting='hard', n_jobs=4)
#LR1 and LR2 are some other Logistic Regression estimators defined in another section
votingC = votingC.fit(X_new_train, y_train_no_id)
votingC.predict(X_new_test)

There are a couple of Github issues on this - see here and here. In short, while scikit-learn native models include an _estimator_type attribute:
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()
clf._estimator_type
# 'classifier'

this is not the case with a KerasClassifier; using your own NN gives
NN = KerasClassifier(build_fn=NeuralNetwork , epochs=100, batch_size=100, verbose=0)
NN._estimator_type

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-4-a2fd193c154f> in <module>()
----> 1 NN._estimator_type

AttributeError: 'KerasClassifier' object has no attribute '_estimator_type'

As reported in a comment here, setting the attribute manually seems to work:
NN._estimator_type = "classifier"

so add this line immediately after your NN definition.


Sentiment Analysis Datasets I am looking for sentiment analysis data, mostly customer product review.

I found a lot of research places provide large size of datasets, but many of them are outdated. I want to get more up-to-date data and I am willing to pay.

Is there any vendor that sells aggregated sentiment data from retailers or e-commerce. Or is there any other avenues we can get the review data.

There is an Amazon Customer Review dataset which you can read more about here - the data is stored in S3 and freely available for anyone to use.

It contains over 160 million customer reviews (the README suggests 130 million, but more data must have been added since that was written) made available by Amazon - you can access them in either TSV or Parquet format.

There is also a blog post on using this data set in sentiment analysis here, which links to another post giving more technical detail. In summary, the blog looks at how sentiment correlates with the star rating in the reviews over time, across product categories, etc.

Welcome to the site! Most people in this scenario use either Yelp dataset and/or the IMDB dataset. The data is polarized enough for you to create sentiment analysis and be able to bootstrap a model from these initial sets to something customized to your particular need.


What is the proper way to bin variables for calculating WoE during credit scoring? I have read this article about developing a credit scorecard in python, where it is stated that when binning the continuous variables, it needs to be ensured that:
1. Each bin should have at least 5% of the observations
2. Each bin should be non-zero for both good and bad loans
3. The WOE should be distinct for each category. Similar groups should be aggregated or binned together. It is because the bins with similar WoE have almost the same proportion of good or bad loans, implying the same predictive power
4. The WOE should be monotonic, i.e., either growing or decreasing with the bins
5. Missing values are binned separately

This seems like a lot of work to accomplish manually (each column needs to be divided in bins, each of these five conditions need to be checked manually, bins should then be adjusted and the conditions need to be checked again). Is there a faster way to do it? Or is there any algorithm/function that bins continuous variables in most practical way.

Or is there any algorithm/function that bins continuous variables in most practical way.

Sure there is, but that's the wrong question: the standard method for discretizing a continuous variable consists in splitting the values into equal intervals, that's it. Of course it doesn't guarantee any of the 5 conditions, since these conditions are about additional constraints almost exclusively related to expert knowledge and the specifics of the data.
Note that these conditions can certainly be automated, there's no need for manual verification. There might be some domain-specific packages which do this for you, but there's no reason a standard ML/statistics library would provide methods for every specific problem like this one.


Hyper-parameter tuning of NaiveBayes Classier I'm fairly new to machine learning and I'm aware of the concept of hyper-parameters tuning of classifiers, and I've come across a couple of examples of this technique. However, I'm trying to use NaiveBayes Classifier of sklearn for a task but I'm not sure about the values of the parameters that I should try.

What I want is something like this, but for GaussianNB() classifier and not SVM:

from sklearn.model_selection import GridSearchCV
C=[0.05,0.1,0.2,0.3,0.25,0.4,0.5,0.6,0.7,0.8,0.9,1]
gamma=[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0]
kernel=['rbf','linear']
hyper={'kernel':kernel,'C':C,'gamma':gamma}
gd=GridSearchCV(estimator=svm.SVC(),param_grid=hyper,verbose=True)
gd.fit(X,Y)
print(gd.best_score_)
print(gd.best_estimator_)

I've tried to search for examples for NaiveBayes, but couldn't find any. What I have right now is simply this:

model = GaussianNB()

What I want is to try different parameters and compare the scores.

from sklearn.model_selection import GridSearchCV

hyper = {'C':[0.05,0.1,0.2,0.3,0.25,0.4,0.5,0.6,0.7,0.8,0.9,1],
         'gamma':[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0],
         'kernel':['rbf','linear']
        }

gd=GridSearchCV(estimator=svm.SVC(),param_grid=hyper,verbose=True)

gd.fit(X,Y)
print(gd.best_score_)
print(gd.best_estimator_)

Sources:

Hyperparameter Tuning of Machine Learning Model in Python

https://github.com/dataprofessor/code/blob/master/python/hyperparameter_tuning.ipynb
https://www.youtube.com/watch?v=AvWfL1Us3Kg

I think you will find Optuna good for this, and it will work for whatever model you want. You might try something like this:

import optuna

def objective(trial):
    hyper_parameter_value = trial.suggest_uniform('x', -10, 10)
    model = GaussianNB(<hyperparameter you are trying to optimize>=hyperparameter_value)

    # evaluate the model here

    return model_accuracy  # or whatever metric you want to optimize

study = optuna.create_study()
study.optimize(objective, n_trials=100)

You can run studies that persist across multiple runs, and you can print out the values of the hyperparameters that worked best, etc.


problem loading data into R I have problem with loading data into R:

fileUrl <- "http://jadi.net/files/iran_it_status_1394_detail_data_jadi_net.tsv"

download.file(fileUrl , destfile="iran_it_status_1394_detail_data_jadi_net.tsv")

dev <- read.delim("iran_it_status_1394_detail_data_jadi_net.tsv",
              header=TRUE,sep="\t",blank.lines.skip = TRUE,
              na.strings="",fileEncoding="UTF-8",
              stringsAsFactors=FALSE,skipNul = TRUE)

I receive the following error:

Error in read.table(file = file, header = header, sep = sep, quote = quote,  :
  no lines available in input
  In addition: Warning message:
  In read.table(file = file, header = header, sep = sep, quote = quote,  :
  invalid input found on input connection    'iran_it_status_1394_detail_data_jadi_net.tsv'

Edit: The dataset has 1217 rows and 33 variables.

names(data) <- c("timestamp","age","sex","birth_province","work_province","experience","education",
            "certificate","learn","project","book","language","wish_language","db","desktop_os",
            "wish_os","mobile","env","theme","src_ctrl","tab_space","drink","items","device","title",
            "org_type","org_emp","income","perk","job_contract","job_type","hour_wage","happy")

for language variable I expect this output:

data[1:3,"language"]

C#, Javascript, R, SQL

Java, C#, Javascript, Objective C, Swift, SQL

C#, SQL

Python Solutions are also welcome

I am able to load the data set like so:

 dev <- read.table("iran_it_status_1394_detail_data_jadi_net.tsv",
                   header=TRUE, sep="\t", blank.lines.skip = TRUE,
                   na.strings="",
                   stringsAsFactors=FALSE, skipNul = TRUE, fill=T, quote="")

Note the removal of the encoding (so that the function "finds" lines in the file), the fill attribute (to allow for a ragged table with empty cells), and the elimination of quotes (apparently there is a misquoted line somewhere near line 585).

This results in a table full of encoded characters - you'll need to know more about the source data to figure out how to work with it, but if you open the file up in a raw text editor (e.g.: Sublime text) you might get some clues:

Try using eithertry or tryCatch:

See here http://stackoverflow.com/questions/13613270/how-to-fix-the-error-in-r-of-no-lines-available-in-input


What's an efficient way to plot many features for EDA in python? I'm trying to visualize data for exploratory data analysis motivated by visualizing  multiple scatterplots of features simultaneously, similar to this question. But I quickly run into problems when using a large number of features (~50) and rows (~50K). While I like using seaborn pairplots the generation of a large number of plot panes can get computational intractable for a large number of features and observations. Subsetting a very large table to a smaller number of features or observations does not seem complete.

My question is: What's an efficient way to plot many features for EDA in python? If there is not an efficient way then is there a defensible way to reduce the number of features or observations

I would suggest t-SNE (just google it). It helps you to have a general overview on what is gling on with your high-dimensional data. Please note that parameters of t-SNE are pretty sensitive so you need to put some effort to find a good embedding.


Why don't we use Manhattan distance instead of euclidean distance in linear regression? When I am explaining concept of linear regression to one of my peers, I got stuck in answer this question. Why don`t we use Manhattan distance instead of euclidean distance in linear regression? Can anyone give intuition behind this?

The main reason for that may be typically use Euclidean metric; Manhattan may be appropriate if different dimensions are not comparable.. While using regression-based methods you may have noticed that you usually have features of real values. You usually normalise the features and feed them to your model. The act of normalising features somehow means your features are comparable. In cases where you have categorical features, you may want to use decision trees, but I've never seen people have interest in Manhattan distance but based on answers [2, 3] there are some use cases for Manhattan too. You can also consider that they are comparable.


How to test trained PCA used for compression? I am working on an exercise for using PCA for compression of images and I don't quite understand how to use it on the test data:

I have 300 images of hand drawn sixes, represented by 28x28 matrices, in the train data, and I have used PCA to find an appropriate low dimensional representation of these images (26 dimensions yields me the sought after 90% threshold), giving me a 300x26 Matrix, that I can use to project my images into that space

Now I have to test this with my test data of 10 similar images - so I have to project them into the same space.

Because I can't just use the train space (dimensions don't agree), if I understood correctly, I should run another PCA to find the Principle Components for these new images, but project them into the 26 dimensions as identified by my train PCA (PCA on just the test suggests that 6 dimensions would suffice, but I want the more accurate 26 from my wide range of training data)

But here is where I'm struggling: how do I centre the test data? To centralise the data before using PCA or the dual PCA, I deduct the mean from the data. When building my test PCA should I deduct the mean of the train data, or the mean of the test data?

Found the answer in a different stackexchange question here: https://stats.stackexchange.com/questions/142216/zero-centering-the-testing-set-after-pca-on-the-training-set

Answer: Yes, zero center the test data with the mean from the train data

You need to combine the datasets first, run the PCA and then split the datasets afterward. Once this is done, you can train your model and then test it. The PCA needs to be done on the whole dataset in order to have the same components in both sets.


What is the difference between and Embedding Layer and an Autoencoder? I'm reading about Embedding layers, especially applied to NLP and word2vec, and they seem nothing more than an application of Autoencoders for dimensionality reduction. Are they different? If so, what are the differences between them?

Actually they are 3 different things (embedding layer, word2vec, autoencoder), though they can be used to solve similar problems. (i.e. dense representation of data)

Autoencoder is a type of neural network where the inputs and outputs are the same but in the hidden layer the dimensionality is reduced in order to get a more dense representation of the data.

Word2vec contains only 1 hidden layer but the inputs are the neighborhood words and the output is the word itself (or the other way around). So it cannot be an autoencoder cause the inputs and outputs are different.

Embedding layer is only a "simple" layer in a neural network. You can imagine it as a dictionary where a category (i.e word) is represented as a vector (list of numbers). The value of the vectors are defined by backpropagating the errors of the network.

Embedding layer, is used in auto-encoders to construct word2vec.

Embedding layer, are a type of layer, used in Deep Learning. You can find others here. Auto-encoders, are a type of architecture, where embedding layers are used. Using these architectures, one can calculate word2vec. word2vec values are calculated when words are fed into the auto-encoders.


What exactly is a skewed dataset ? Like above, I'd like to know, what exactly a skewed dataset is.

The explanation from stats.se.com sounds to me more like what I call an imbalanced dataset?

What is the distinction?

In the context of the link, a skewed data set is referring to a dataset with a class imbalance problem. They are trying to build a classifier, but they have many more negative examples than positive examples. It's not a very precise term, but I've heard to used in this context a few times.

Skewed classes are scenarios where the number of observations belonging to one class are significantly lower than those belonging to other class. These occur commonly in electricity pilferage, fraudulent bank transaction etc. There are many techniques which can help to reduce this non-uniformity like SMOTE etc.


What is the "dying ReLU" problem in neural networks? Referring to the Stanford course notes on Convolutional Neural Networks for Visual Recognition, a paragraph says:

  "Unfortunately, ReLU units can be fragile during training and can
  "die". For example, a large gradient flowing through a ReLU neuron
  could cause the weights to update in such a way that the neuron will
  never activate on any datapoint again. If this happens, then the
  gradient flowing through the unit will forever be zero from that point
  on. That is, the ReLU units can irreversibly die during training since
  they can get knocked off the data manifold. For example, you may find
  that as much as 40% of your network can be "dead" (i.e. neurons that
  never activate across the entire training dataset) if the learning
  rate is set too high. With a proper setting of the learning rate this
  is less frequently an issue."

What does dying of neurons here mean?

Could you please provide an intuitive explanation in simpler terms.

A "dead" ReLU always outputs the same value (zero as it happens, but that is not important) for any input. Probably this is arrived at by learning a large negative bias term for its weights.

In turn, that means that it takes no role in discriminating between inputs. For classification, you could visualise this as a decision plane outside of all possible input data.

Once a ReLU ends up in this state, it is unlikely to recover, because the function gradient at 0 is also 0, so gradient descent learning will not alter the weights. "Leaky" ReLUs with a small positive gradient for negative inputs (y=0.01x when x < 0 say) are one attempt to address this issue and give a chance to recover.

The sigmoid and tanh neurons can suffer from similar problems as their values saturate, but there is always at least a small gradient allowing them to recover in the long term.

The "Dying ReLU" refers to neuron which outputs 0 for your data in training set. This happens because sum of weight * inputs in a neuron (also called activation) becomes <= 0 for all input patterns. This causes ReLU to output 0. As derivative of ReLU is 0 in this case, no weight updates are made and neuron is stuck at outputting 0.

Things to note:

Dying ReLU doesn't mean that neuron's output will remain zero at the test time as well. Depending on distribution differences this may or may not be the case.
Dying ReLU is not permanent dead. If you add new training data or use pre-trained model for new training, these neurons might kick back!
Technically Dying ReLU doesn't have to output 0 for ALL training data. It may happen that it does output non-zero for some data but number of epochs are not enough to move weights significantly.


How to draw Deep learning network architecture diagrams? I have built my model. Now I want to draw the network architecture diagram for my research paper. Example is shown below:

I recently found this online tool that produces publication-ready NN-architecture schematics. It is called NN-SVG and made by Alex Lenail.

You can easily export these to use in, say, LaTeX for example.

Here are a few examples:

AlexNet style

LeNet style

and the good old Fully Connected style

We can use Powerpoint to get the job done.

Draw the diagram (3D rectangles and perspectives come handy) -> select the interested area on the slide -> right-click -> Save as picture -> change filetype to PDF -> :)


xgboost Predictions from R and Python don't match Thought maybe somebody here could help us solve a mystery (https://github.com/dmlc/xgboost/issues/1623):

We are trying to build a xgboost prediction function in R for a model that was trained in Python and the results don't match. See below for an example of how to reproduce.

Fairly new to xgboost, particularly using it across languages, so may be missing something obvious.

Steps to reproduce:

(1) Download this model file: http://ml.stat.purdue.edu/hafen/WTKG.model

(2) Run this script in R:

library(xgboost)
mod <- xgb.load("WTKG.model")
x <- c(91, 9, 9, NA, NA, 273, 20, 170, NA, NA, 14, 14, 0,
  2, 0.94289404091, 0.94289404091, 0.93087973569, 0.0120143052199997, 0.95490834613,
  0.95490834613, 1, 90, 0.95490834613, 1, 90,
  0.93087973569, 357, -266, 0.93087973569, 357, -266,
  0.95490834613, NA, 0.93087973569, NA, NA, NA)
d <- xgb.DMatrix(matrix(x, nrow = 1), missing = NA)
predict(mod, d)
# [1] 0.6483372

(3) Run this script in Python:

import numpy as np
import xgboost as xgb

bst = xgb.Booster({'nthread': 4})
bst.load_model('WTKG.model')
x = [91, 9, 9, np.nan, np.nan, 273, 20, 170, np.nan, np.nan, 14, 14, 0,
  2, 0.94289404091, 0.94289404091, 0.93087973569, 0.0120143052199997, 0.95490834613,
  0.95490834613, 1, 90, 0.95490834613, 1, 90,
  0.93087973569, 357, -266, 0.93087973569, 357, -266,
  0.95490834613, np.nan, 0.93087973569, np.nan, np.nan, np.nan]
d = xgb.DMatrix(data=[x], missing=np.nan)
bst.predict(d)[0]
# 1.3775804

Solution for posterity: In Python, xgboost requires np.array as input. So this line:

d = xgb.DMatrix(data=[x], missing=np.nan)

should be:

d = xgb.DMatrix(data=np.array([x]), missing=np.nan)

I think you can get your answer from here


Categorical data in Kmeans I need to perform clustering in a given dataset.
There are atrributes with numerical as well as categorical values.

What is the best way to convert categorical to numeric value ?

as an example one field is colour and the values are red, green, blue
so can I assign a  mapping like:

red : 1, green : 2, blue : 3 or
red : 11, green : 25, blue : 30

and if I provide a mapping like this will this affect the Euclidian distance for clustering ?

or is there any other way ?

Converting the categorical data into numerical data isn't really meaningful. Different mappings will give you different solutions.

There is an extension of K-means algorithm for categorical data called k-modes. You can read about K-modes in detail here. This article explains the difference between K-modes and converting data into numeric vectors and then running K-means.

Try to categorize them as efficiently based on what you care about.

If you want to stay true to the color wheel than just use the RGB values for the colors you are using.

However, if not then you can use any mapping, but be careful because it will affect your Euclidean distance even after you normalize the feature (which is highly recommended for K-means).


Generate new images based on a dataset of images Is it possible take a training set of one million 120 x 120 pixel tiles, feed these tiles into a machine learning algorithm. And then make it synthesize images that look like the original training set with a certain, adjustable level of entropy?

What ML techniques should I consider?

Yes, there are plenty ways to do this, what you basically want is to interpolate between the two.

This is current state-of-the-art for solving problems like yours: https://github.com/carpedm20/BEGAN-tensorflow

Also keep an eye out for my question here: 2D Image-based Avatar Animation With Expressions / Image-To-Image Translation

If I end up using the ML approach, you could very well also use it with tiles instead of faces and be able to adjust the likeness. I will link to my project in my question.


Can a recommendation system be built without any user ratings? I was planning to make an artwork recommendation system as a project by using the WikiArt open source dataset available on kaggle, I'm still looking for datasets which might already have user ratings but is it possible to make a recommender without any user ratings? The dataset currently has these features. Any help or insight is appreciated. Thanks.

This is a common issue for recommender system refered to as cold start. You can look it up there are some solutions suggested out there but I have never tested them out. Have a look at this paper:

http://www.cs.toronto.edu/~mvolkovs/nips2017_deepcf.pdf

First you need to clarify what kind of recommendation you want to make.

The reason for the rating is to be able to associate an item to a user preference. You can avoid a rating and design the recommendation as a classification problem where you classify a product to some event (someone would like the artwork or not), but this still needs you to design a label based on some information.

In summary yes you can do without a rating but you still need some information on user preference or whatever the label will be.


How to treat the undefined values which make sense? I'm currently trying to create a few features to improve the performances of a model. One of those features that I would like to create corresponds to the difference in days between a customer's purcharse and his last one. To create this feature is not a problem. However, I don't know which value to set if this is the first purcharse of a customer. Which value should I set and, more generally, how to treat these cases ?

   customer_id date_purchase  diff_last_purchase  first_purchase
0            1    2018.02.12                 NaN               1
1            1    2018.02.18                 6.0               0
2            2    2018.02.25                 NaN               1
3            3    2018.03.15                 NaN               1
4            3    2018.03.18                 3.0               0

Recently had a discussion on the same topic at work. It boiled down to encoding missing values as impossible values (negatives, very high) or as information inferred from the dataset (mean, median). Some more sophisticated methods use models build on rest of data (non-missing columns) to predict missing ones.

If using tree-based approach setting it to -1 should be fine as a start, as as there might be observations where previous purchase was in same day (0).

With mean-trending models (linear regression), setting it to the mean might be also fine, but you need to calculate the mean on the train set only and propagate it to the test set separately.

In general, if we don't know the reason of missing data, it's hard to treat them properly. This reason can highly affect our conclusion. So my first recommendation is always trying to figure out why the data is missing in the first place.

Normally, there are 3 kinds of missing data: (Definitions on Wikipedia)

Missing completely at random

  Values in a data set are missing completely at random (MCAR) if the events that lead to any particular data-item being missing are independent both of observable variables and of unobservable parameters of interest, and occur entirely at random. When data are MCAR, the analysis performed on the data is unbiased; however, data are rarely MCAR.

Missing at random

  Missing at random (MAR) occurs when the missingness is not random, but where missingness can be fully accounted for by variables where there is complete information. Since MAR is an assumption that is impossible to verify statistically, we must rely on its substantive reasonableness.

Missing not at random

  Missing not at random (MNAR) (also known as nonignorable nonresponse) is data that is neither MAR nor MCAR (i.e. the value of the variable that's missing is related to the reason it's missing).

Also from that Wikipedia page:

  Missing data reduces the representativeness of the sample and can therefore distort inferences about the population. Generally speaking, there are three main approaches to handle missing data: (1) Imputation—where values are filled in the place of missing data, (2) omission—where samples with invalid data are discarded from further analysis and (3) analysis—by directly applying methods unaffected by the missing values.

"In the case of MCAR, the missingness of data is unrelated to any study variable", so you can just drop them or do some reasonable imputation and an continue to analyze the data. I know only one test to test MCAR, which is Little's Test.

I don't know how to deal with the second and third cases, so all I can say is please treat them with care. If the ratio of missing data is too large, I simply drop the whole dataframe.


Feature reduction convenience In the field of machine learning, I'm wondering about the interest of applying feature selection techniques.

I mean, I often read articles or lectures speaking about how to reduce the number of feature (dimensionality reduction, PCA), how to select the best features (feature selection etc).

I'm not sure of the main purpose of this:

Does feature reduction techniques always improve accuracy of the learned model?
Or is it just a computational cost purpose?

I would like to understand when it is necessary to reduce the number of features and when it is not, in order to improve interpretability or accuracy.
Thanks!

Feature Selection (FS) methods are focused on specializing the data as much as possible to find accurate models for your problem. Some of the main issues that drive the need for FS are:

Curse of dimensionality: Most algorithms suffer to grasp relevant chacteristics of data for a specific prediction task, when the number of dimensions (features) of the data is high, and  the number of examples is not sufficiently big. Check here some more detailed explanation
Correlation between variables: Typically, the presence of highly correlated pair of variables can cause ML algorithms to pay to much attention to a particular effect that is "over-represented". For this reason many FS methods address the reduction of this correlation. Reducing the number of correlated variables oftenly increases the model predictive power.
Latent features: Although specific variables might be highly expressive for your problem, a lot of power can be achieved when finding "latent features", such as linear and non-linear combinations of the original variables. Here there are hundreds of approaches, from PCA to Neural Networks. Independently from the approach (and statistical assumptions) the idea is to create new features that condense the information of a bigger set of features into a smaller one. Hopefuly the new set of features is more representative, and being smaller could be more easily learnt.

Feature selection does not necessarily improve the predictive quality of the model. Reducing or transforming the features might lead you to loss of information and then a less accurate model. Is an open and complex field of research. However in many cases it becomes quite useful. It will depend on how good and different are your original features for describing the target variable.
If you look into bionformatics, you'll see people dealing with thousands or even millions of features, while having only hundred of examples. Here feature selection becomes increasingly relevant.

PS: Most commonly I've seen the term "feature selection" used for the creation of compound features, as most examples I've mentioned, while Feature Extraction term is used for actually removing specific features from the dataset without taking into account is relation with the target variable.

Feature selection could be done to prevent overfitting. The more features you have, the more likely you'll overfit. And yes, I'd say one of the purposes could be to reduce computational costs.


Simple Explanation of Apache Kafka Can anybody explain Apache Kafka for me in a plain language? I'd appreciate an explanation with a practical example instead of abstract theoretical definitions, then I can understand better.

What is it used for? What does messaging mean? Messaging between what?!! At which stage of a BigData analysis is it used?

And what are prerequisites for learning it?

PS:
Please explain as you would explain for a non-technical person

Can anybody explain Apache Kafka for me in a plain language?

As the official documentation of Kafka says:

  Apache Kafka is publish-subscribe messaging rethought as a distributed
  commit log. It follows the publish-subscribe messaging style, with speed and durability built in.

Now, let me give a dumbed down explanation of the same:

A publish-subscribe(commonly called pub-sub) architecture basically contains a publisher(who sends the messages, for example: A twitter stream) and a subscriber(who receives the messages Ex: the analyst/the user's activity stream)

A clearer explanation would be:

  In software architecture, publish–subscribe is a messaging pattern
  where senders of messages, called publishers, do not program the
  messages to be sent directly to specific receivers, called
  subscribers, but instead characterize published messages into classes
  without knowledge of which subscribers, if any, there may be.
  Similarly, subscribers express interest in one or more classes and
  only receive messages that are of interest, without knowledge of which
  publishers, if any, there are.

And keep in mind that Kafka is a distributed pub-sub messaging system, designed to scale.

Now,

  At which stage of a BigData analysis is it used?

It is basically used in the Extraction step of the ETL pipeline. It can contain high volumes and speed of data flow (we call it high throughput in technical terms).

It can store messages also, but it is a persistent store; which means that the data is not stored forever. The stored data has an expiry date.

  And what are prerequisites for learning it?

This is a tough question, as everyone who can program can get started with it, but if you really want to implement this in your architecture, then these are the pre-requisites which I can think of:

Know what is a persistent data store and where to use them
Know what a pub-sub messaging system is
Know what an extraction step is, in ETL. And also know how to deal with data with huge volume coming at you at high velocity.

Further reading:

Kafka has an excellent documentation, which can help to get started with it.

It might be easier for you to take a look at the events streaming services that are offered in Amazon Web Services (AWS), mainly Kinesis.

The easiest of them in Kinesis Firehose (http://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html), which gives you an endpoint where you can write your events to, and they will appear in Amazon Redshift, which is a data warehouse service that can scale to huge sizes (https://aws.amazon.com/redshift/).

AWS also announced that they will add Kinesis Analytics, which will allow you to write SQL code on top of the events stream of Kinesis. You can calculate different statistics such as moving agerages and other relevant statistic on your incoming events.

These events that are processed in near real time, can be clicks on your web site, purchases in your e-commerce site, security events that should be monitored for potential intrusion, etc.