Train/Test/Validation Set Splitting in Sklearn How could I split randomly a data matrix and the corresponding label vector into a X_train, X_test, X_val, y_train, y_test, y_val with Sklearn? As far as I know, sklearn.cross_validation.train_test_split is only capable of splitting into two, not in three...

You could just use sklearn.model_selection.train_test_split twice. First to split to train, test and then split train again into validation and train. Something like this:

 X_train, X_test, y_train, y_test
    = train_test_split(X, y, test_size=0.2, random_state=1)

 X_train, X_val, y_train, y_val
    = train_test_split(X_train, y_train, test_size=0.25, random_state=1) # 0.25 x 0.8 = 0.2

import numpy as np
import pandas as pd

#length of data
N = 10
scale=2

#generated random data
X, y = np.arange(N*scale).reshape((N, scale)), np.arange(N)

#Works for pandas dataframe too
#You can download titanic.csv from here
#https://github.com/fuwiak/faster_ds/blob/master/sample_data/titanic.csv

#df = pd.read_csv("titanic.csv", sep="\t")
#X=df[df.columns.difference(["Survived"])]
#y=df["Survived"]

def train_test_val(X, y, train_ratio, test_ratio, val_ratio):
    assert sum([train_ratio, test_ratio, val_ratio])==1.0, "wrong given ratio, all ratios have to sum to 1.0"
    assert X.shape[0]==len(y), "X and y shape mismatch"

    ind_train = int(round(X.shape[0]*train_ratio))
    ind_test = int(round(X.shape[0]*(train_ratio+test_ratio)))

    X_train = X[:ind_train]
    X_test = X[ind_train:ind_test]
    X_val = X[ind_test:]

    y_train = y[:ind_train]
    y_test = y[ind_train:ind_test]
    y_val = y[ind_test:]

    return X_train, X_test, X_val, y_train, y_test, y_val
# put ratio as you wish
X_train, X_test, X_val, y_train, y_test, y_val=train_test_val(X, y, 0.8, 0.1, 0.1)


difference between XGBRegressor and XGBClassifier I'm trying to understand the difference between xgboost.XGBRegressor and  xgboost.sklearn.XGBClassifier.

Can someone explain the difference in a concise manner?

Because when I fit both classifiers with the exact same data, I get pretty different performance.

This is how I fit the data.

clf = xgboost.XGBRegressor(alpha=c)
#clf = xgboost.sklearn.XGBClassifier(alpha=c)

clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
print('Model roc auc score: %0.3f' % roc_auc_score(y_test, y_pred, average='macro', sample_weight=None))
print('Model pr auc score: %0.3f' % average_precision_score(y_test, y_pred))

when  clf = xgboost.sklearn.XGBClassifier(alpha=c)

Model roc auc score: 0.544

Model pr auc score: 0.303

when clf = xgboost.XGBRegressor(alpha=c)

Model roc auc score: 0.703

Model pr auc score: 0.453

What would cause this performance difference?

XGBRegressor is for continuous target/outcome variables. These are often called "regression problems."

XGBClassifier is for categorical target/outcome variables. These are often called "classification problems."

It is not like that..xgbregreesor can also be used for classifier with with objective = "binary:logistic" . I found xgbregreesor to give higher auc than xgclassifier.Though I am not expert in it..you can try both and which gives higher accuracy use that


Why the number of neurons or convolutions chosen equal powers of two? In the overwhelming number of works devoted to the neural networks, the authors suggest arhitechure in which each layer is a numbers of neurons is power of 2

what are the theoretical reasons(prerequisite) for this choice?

Deep Neural Networks are usually trained on GPUs to speed up training time. Using power of two for the network topology follows the same logic as using power of two for image textures in computer games.

The GPU can take advantage of optimizations related to efficiencies in working with powers of two. (see https://gamedev.stackexchange.com/questions/26187/why-are-textures-always-square-powers-of-two-what-if-they-arent)

It is just an arbitrary choice. You have to choose one number and the order of magnitude matters, but not the exact value. Powers of two just feel natural.

If you don't think so: Evaluate it on a given architecture. Lower the number of neurons from a power of two to a smaller number. If the time increases, you've proven me wrong.


How to use scikit-learn normalize data to [-1, 1]? I am using scikit-learn MinMaxScaler() to normalize to $[0, 1]$, but I want to normalize to $[-1, 1].$ What function do I have to use to normalize this way?

Scaling between 0 and 1 is simply written for an array of values arr = $[x_{1}, x_{2}, ...., x_{n}]$ as scaled_array = (arr-arr.min())/(arr.max()-arr.min())

But scaling between two values can be inherited from normalisation in Images when values are scaled between 0 to 255 (normally).

You can write scaling as scaled_values = ((val - min)*(new_max - new_min)/(max - min)) + new_min

where new_min, new_max are the maximum and minimum values you are scaling in between and max and min are the maximum and minimum values in your array.

For example if you want to scale values between -1 and 1 for an array [2, 5, 0], it reduces to ((arr - 0)*(1-(-1))/(5-0)) - 1 = [-0.2, 1, -1]

Check this wiki page.

Edit:

As pointed out rightly in comments by @AnkitSeth, MinMaxScalar do have feature_range argument you can pass on. But I am sure the internal job done by feature_range is the same thing as I had mentioned above.

Try using StandardScaler() from sklearn.preprocessing.

This is the proper normalization. But we cannot give any range.
Only mean is zero, that is for sure.


Time series analysis on small dataset I am new to time series and therefore trying to feel my way around. I am required to do some analysis and possibly come up with a model. However, I have at most 2 years worth of quarterly data (at most 8 datapoints). Can I build any meaningful model based on the data? What possible model apart from linear regression with time as regressor can be built?

Thanks

There is no valid answer. If you have a stationary process with 0 variance, then the forecast horizon has no limit.

More realistically, you have the follow rule of thumb (which is totally from experience with absolutely no theoretical base): the horizon forecast may be half the historical base. You have 8 data points, so you can forecast 4 points. You have 2 years, so you can forecast 1 year.

The key point is : check this after your forecast is done.

I would start playing around with the prophet library for python, you can still make predictions with 8 datapoints, the accuracy is going to depend on a lot of things especially the variance in the points.


Constructing function - f(x,y) for the given minimums (Python) Problem Statement:
I need to construct a function f(x,y) in which there're 3 minimums. 2 local and 1 global which are written below.

Locals are: z = f(0.2,0.3) = 0.7 | z = f(0.6,0.8) = 0.8
Global is: z = f(0.85,0.5) = 0.6

What I've tried so far
I've tried Polynomial regression to construct features that estimate the given minimum points. However, an estimated function z = f(x,y) just going through all the given minimum points without considering them as minimums. I need an approach by means of which I can construct a function that has 3 minimums that are mentioned above. I'd be more than happy if you could provide a code snippet of your approach. As a programming language, I use Python.
Code
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D

degree = 6

def func(x1, x2):
    arr = []
    for i in range(degree, 0, -1):
        arr.append(x1 ** i)

    for i in range(degree, 0, -1):
        arr.append(x2 ** i)

    arr.append(1)

    return np.array(arr)

def der_func(x1, x2):
    arr1 = []
    arr2 = []
    for i in range(degree - 1, -1, -1):
        arr1.append((i + 1) * x1 ** i)

    for i in range(degree + 1):
        arr1.append(0)

    for i in range(degree):
        arr2.append(0)

    for i in range(degree - 1, -1, -1):
        arr2.append((i + 1) * x2 ** i)

    arr2.append(0)

    return np.array([arr1, arr2])

given_x = np.array([[0.2, 0.3],
                    [0.6, 0.8],
                    [0.85, 0.5]])

given_y = np.array([0.7, 0.8, 0.6])

data_x = []
data_y = []

for i in range(len(given_x)):
    x1x2 = given_x[i]

    data_x.append(func(x1x2[0], x1x2[1]))
    data_y.append(given_y[i])

    if i < 3: # in case there's more than 3 data point in given_x
        data_x.append(der_func(x1x2[0], x1x2[1])[0])
        data_y.append(0)

        data_x.append(der_func(x1x2[0], x1x2[1])[1])
        data_y.append(0)

data_x = np.array(data_x)
data_y = np.array(data_y)

w = np.linalg.inv(data_x.T @ data_x) @ data_x.T @ data_y
pred_y = data_x @ w

cost = np.mean((data_y - pred_y) ** 2)
print("Cost | ", cost)

#######################################

X = np.linspace(0, 1, 100)
Y = np.linspace(0, 1, 100)

X, Y = np.meshgrid(X, Y)
Z = np.zeros(np.shape(X))
for i in range(len(X)):
    for j in range(len(X[0])):
        Z[i, j] = np.array([func(X[i, j], Y[i, j])]) @ w

fig = plt.figure()
ax = fig.gca(projection='3d')
ax.contour(X, Y, Z, 150)
ax.scatter(given_x[:3, 0], given_x[:3, 1], given_y[:3])
plt.show()

One way to go is to define functions that are non-zero only near your 'minimal' points, and add them. The goal is to avoid overlap, such that a function associated with a point won't modify the value of your global function around other points.

import numpy as np

def func0(x1):
    lim = 0.01
    dist = x1*x1
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

That will give you a 1D function with nice properties (value 1 at 0, 0 outside the neigborhood of 0) :

x = np.linspace(-1, 1, 1000)
z = np.array([func0(i) for i in x])
Z = z.reshape(1000)

plt.plot(Z)
plt.show()

Modify it to get a 2 D function centered in (a,b) :

def func(x1,x2):
    a = 0
    b = 0
    lim = 0.01
    dist = (x1-a)*(x1-a) + (x2-b)*(x2-b)
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

This will give you a 2d function :

x = y = np.linspace(-1, 1, 100)
z = np.array([func(i,j) for j in y for i in x])
Z = z.reshape(100, 100)

x, x = np.meshgrid(x, y)

plt.imshow(Z,extent=(x.min(), x.max(), y.min(), y.max()))
plt.colorbar()
plt.show()

Then build the threes functions associated with your points.

def func1(x1,x2):
    a = 0.2
    b = 0.3
    lim = 0.01
    dist = (x1-a)*(x1-a) + (x2-b)*(x2-b)
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

def func2(x1,x2):
    a = 0.6
    b = 0.8
    lim = 0.01
    dist = (x1-a)*(x1-a) + (x2-b)*(x2-b)
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

def func3(x1,x2):
    a = 0.85
    b = 0.5
    lim = 0.01
    dist = (x1-a)*(x1-a) + (x2-b)*(x2-b)
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

And add them (up to coefficients that determine the value on points of interest) to form a global function :

def global_func(x,y):
    return 1-0.3*func1(x,y)-0.2*func2(x,y)-0.4*func3(x,y)

This will give you :

x = y = np.linspace(0, 1, 1000)
z = np.array([global_func(i,j) for j in y for i in x])
Z = z.reshape(1000, 1000)

x, x = np.meshgrid(x, y)

plt.imshow(Z,extent=(x.min(), x.max(), y.min(), y.max()))
plt.colorbar()
plt.show()

Finally you can check that :

print(global_func(0.2,0.3))
print(global_func(0.6,0.8))
print(global_func(0.85,0.5))

returns :

0.7
0.8
0.6

You have a continuous function with all expected properties by construction.

If you are unhappy with it being constant on a lot of the domain, you can modify the individual functions so that they don't stop at the limit with :

value = np.exp(-dist)

And in your global function, determine which 'minimal' point is the nearest and only apply a function of the form 1-coef * funci based on which point is nearest. This way the individual functions won't overlap and thus won't modify value of the global function around other 'minimal' points.

Add as additional constraint that first derivative is 0, and that its bigger/smaller than all the values on the interval implying maximum or minimum.


ROC curve interpretation I trained a CNN model and a combined CNN-SVM model for classification. I wanted to compare their performance using ROC curve but I was confused which model is better. How to interpret the given ROC curves ?

If you hear from the area under the curve (AUC), you can find that the first classifier is better as AUC of the first curve is more than the second classifier. To know more about AUC you can find this post useful.

One additional view point i would like to add is that AUC is a ranking metric (what matters is the score order but not the score value itself). so if you order your scores before evaluating it with roc auc metric you can boost your score. Read more about here https://towardsdatascience.com/an-interesting-and-intuitive-view-of-auc-5f6498d87328


Keep track of trainings, datasets eetc After searching quite some time for it on Google I could not find a sufficient software/toolbox that can manage trainings of neural networks. I thought of a program that combines visualization techniques without the need to write code as well as having the possibility to compare several trainings of neural networks and be able to store them easily.
Does a program like this exist?

Regards
Lukas

It does, it's an API called Keras, written on top of TensorFlow.
With that you can rapidly prototype any "standard" NN architecture, such as feedforward NN, RNN, or CNN. With it you can export the trained model with one line of code and save it for later use. As for visualization, Keras offers integration with pydot (GraphViz) to plot the whole NN as a graph for you to inspect.

After further searching online I found these useful packages:

sumatra
sacred
FGLab
CDE


Count the max number of consecutive 1 and 0 in Pandas Dataframe Hey I have the following Dataset
import pandas as pd
df = pd.DataFrame({
'column1': [0,0,1,0,1,0,0,1,1,0,1,1,1]})

I want to be able to count the number of consecutive 1 and 0 and generate 2 columns as such:
consec0: 1,2,_,1,_,1,2,_,_,1,_,_,_
consec1: _,_,1,_,1,_,_,1,2_,1,2,3

I then want to take the max number of consecutive and create two lists:
max_consec0: 2,1,2,1
max_consec1: 1,1,2,3

My dataset in the end will be just max_consec0 and max_consec1

To check if a value has changed, you can use .diff and check if it's non-zero with .ne(0) (the NaN in the top will be considered different than zero), and then count the changes with .cumsum, like this:
df['counter'] = df.diff().ne(0).cumsum()

Afterward, you can create a second dataframe, where the indices are the groups of consecutive values, and the column values are the value (0 or 1, in your case) and length (which is what you ultimately want):
df2 = df.groupby('counter')['column1'].min().to_frame(name='value').join(
df.groupby('counter')['column1'].count().rename('number'))

The resulting max_consec0, max_consec1 are just the values in the [number] column, filtered by the [value] column:
max_consec0 = df2[df2['value']==0]['number'].tolist()
max_consec1 = df2[df2['value']==1]['number'].tolist()

You can verify that the result is [2, 1, 2, 1] and [1, 1, 2, 3], as desired.

You can try this implementation:
num0=0
num1=0
consec0=[]
consec1=[]
for i in range(len(df)):
  if(df.iloc[i,0])==0:
    num0=num0+1;
    num1=0;
  if(df.iloc[i,0])==1:
    num0=0;
    num1=num1+1;
  consec0.append(num0)
  consec1.append(num1)
df['consec0']=consec0
df['consec1']=consec1
max_consec0=df['consec0'].max()
max_consec1=df['consec1'].max()


Neural Network Architecture for Identifying Image Copies I have a large image collection and wish to identify the images within that collection that appear to copy other images from the collection.

To give you a sense of the kinds of image pairs that I wish to classify as matches, please consider these examples:

I have hand classified roughly .25M pairs of matching images, and now wish to use those hand labelled matches to train a neural network model. I am just not sure which architecture would be ideally suited for this task.

I originally thought a Siamese Network might be appropriate, as they have been used for similar tasks, but the output from those classifiers seems more ideally suited to finding different figurations of the same object (which is not what I want), rather than different printings of the same figuration (which is what I want).

If anyone can help recommend papers or architectures ideally suited to identifying images given the training data I have prepared, I would be tremendously grateful for any insights you can offer.

U need to read about triplet loss function. Triplet loss function gets result embeddings from a network, that process 3 images by a network (2 similar and 1 non-similar) for one step:

After that loss computing as:

For more details read the paper from triplet loss authors.

Also may help PSNR, but this is not Deep Learning.

If the images are more similar like you posted, you can go with Structural similarity index which gives output in the range -1 to 1. any thing more than 0.9 can be considered similar.


Amazon SageMaker input data? I am exploring Amazon SageMaker as a scalable machine learning solution.

My question is; is it required that the training data first be uploaded in Amazon S3?

Yes, the built-in models pull from S3 and write the computed model back to S3.


Question about (Python/Orange) Apriori associative algorithm I'm trying to wrap my head around Association rules and frequent itemsets.
So I threw my data in, instead of the samples one and sometime it works, sometimes it doesn't.

rules = Orange.associate.AssociationRulesSparseInducer(data, support = 0.3)

print "%5s   %5s" % ("supp", "conf")
for r in rules:
    print "%5.3f   %5.3f   %s" % (r.support, r.confidence, r)

inducer = Orange.associate.AssociationRulesSparseInducer(support = 0.2, store_examples = True)
itemsets = inducer.get_itemsets(data)
print itemsets
print data.domain
print [data.domain[i].name for i in itemsets[4][0]]

More often than not, itemsets[4][0] shows an IndexError: list index out of range error.
So I start playing around with support = 0.3, support = 0.5, support = 0.2 and itemsets[2][0] or itemsets[3][0].

From the docs:

  support
  Minimal support for the rule. Depending on the data set it should be set to sufficiently high value to avoid running out of working memory (default: 0.3).

True - I tried 0.2 and it quickly blasted my memory on a 800 rows data file.

Any idea what I should do best there or which are viable values for a shopping cart analysis?

800 rows of data (800 orders)
1 to x item(-categories) per order
15 different item-categories in the file, so my data looks like:

ItemCat1
ItemCat2, ItemCat2, ItemCat2, ItemCat2, ItemCat7, ItemCat7, ItemCat7,     ItemCat7, ItemCat7
ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1,     ItemCat2
ItemCat4, ItemCat4
ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1, ItemCat1,     ItemCat1, ItemCat2
ItemCat5

If Python 3 is an option, Orange 3 features FP-growth in its Orange3-Associate add-on:

pip3 install Orange3-Associate

It's well documented: http://orange3-associate.readthedocs.org/

Apriori due to its slowness and resource requirements, as you'd noticed, really isn't an algorithm to consider anymore.

It is not available in Orange, but i would try FP-Growth. Its way faster in combination with low support values than apriori!


Upper bound on size of sample set for decision trees Say I have an instance space with 4 features and I know that a decision tree with 8 nodes can represent the target function I want to learn. I want to give an upper bound on the size of the sample set needed in order to achieve a true error of at most x%.

I found this theorem in a text book.

  Let $\mathcal{H}$ be an hypothesis class and let $\epsilon, \delta > 0$. If a
  training set $S$ of size

  $$n\geq\frac{1}{\epsilon}\operatorname{ln}\left(\frac{|\mathcal{H}|}{\delta}\right)$$

  is drawn from distribution $\mathcal{D}$, then with probability greater than or equal to $1-\delta$ every $h \in \mathcal{H}$ with true error $err_D(h) \geq \epsilon$ has training error $err_S(h) > 0$. Equivalently, with probability greater than or equal to $1-\delta$, every $h \in \mathcal{H}$ with training error zero has true error less than $\epsilon$.

My question is, can I use this theorem to give such an upper bound? If yes, what would the size of my hypothesis space $|\mathcal{H}|$ be in this case? If not, how can I give this upper bound?

As far as I understand this theorem (not much), it doesn't really say anything about the relation between $n$ (the size of the training set) and the error rate. The main relation that it claims is between the error on the training set and the true error, assuming a minimum size on the size of the training set and that the training set is drawn from the true distribution.

I don't think it's possible to give an absolute upper bound on the number of samples needed in the training set for at least the following reasons:

The success of the learning depends on which instances are in the training set, i.e. one needs a sample representative enough so that the model correctly captures the distribution. For instance, in theory it could happen by chance that all the $n$ instances have the same value for feature $x$, so the model would never know that feature $x$ can have another value. In general there's no way to be sure that the training set contains all the diversity required for a successful model in the instances.
There's also the question of what the learning algorithm does exactly: if one wants to prove that a particular DT learning algorithm (say C4.5) can learn a particular model with less than $n$ instances (assuming these $n$ instances are sufficient), the proof must involve the particular properties of the algorithm itself. Otherwise it's obvious that the claim is false: if one chooses an algorithm which randomly builds a tree, it's clear that this cannot lead to the right model.

I think it's more feasible and more common to provide an experimental upper bound: run many cross-validation experiments, varying the training set size and possibly other parameters. Then based on this one can estimate the probability of reaching the correct model given the size and other parameters.


Does the SVM require lots of features most of the time? So I know about the curse of dimensionality (too many features too less data).

Say I have a 3000 sample dataset, would 3 features be too less?

I stumbled upon some rules of thumb for dataset sizes, but not specific to the ratio of features / samples.

Also, it seems easier to guess to the "positive" side, i.e. "might it work?" then "might it fail?".
I witnessed both small datasets (<1000) that functioned well with even 20-50 features, and huge datasets (~10M) that functioned well with very few (<5) features.

The stage in the processing pipeline to tackle this, is the feature selection one. I'd start with gathering all of your make-sense features and playing with different k-best values for the feature selection object to find. (ranges like: 10, 50, 100, ..)

You didn't mention a specific research environment, but I'd like to suggest scikit-learn sklearn.feature_selection.SelectKBest running inside a GridSearchCV pipeline. This way the grid object will help you choose the closest k-best featues out of a given list. Something like:

from sklearn.feature_selection import SelectKBest
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.svm import LinearSVC

pipe = []
pipe.append(SelectKBest())
pipe.append(LinearSVC())

est_pipe = make_pipeline(*pipe)
print 'all possible grid params: {0}'.format(est_pipe.get_params().keys())
param_grid = {
  # some svc grid params..,
  'selectkbest__k': [5, 10, 50, 100]
}

grid = GridSearchCV(est_pipe, param_grid=param_grid)
grid.fit(X, y)

So I'll post an answer to my own question. For anyone who comes across this post during the feature selection / More features or less process, I dont know what you can do (well except if you're on python then mork's answer has a good way to do feature selection there) but I can tell you what NOT to do.

Do not under any circumstances ever "try" determining best features by training+testing the SVM / statistical model. That is oh this feature works because of more classification accuracy than the other one. NO. Not unless that is the only way left, dont do it. That is a way, you are free to do it, but if you can try something else please do. Dont listen to anyone who tells you to do that

How many features your problem requires depends on how many optimal features you can find. I'll leave it at that. How to find them? That is the million dollar question.

Edit:

People are getting confused. When you dont know about the accuracy of your features, it is bad practice to "train" data to see how many features your SVM needs. For that it is better you select features on the basis of some criteria set by your problem. If you want after that, then you may try feature selection techniques. But remember, reducing too many dimensions may also decrease accuracy sometimes.


How can conclusions be drawn from recommendation systems evaluation? From my research, a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And basically exists many type of recommendation systems, such as collaborative filtering and content-based.

One important aspect, is that evaluation is important in assessing the effectiveness of recommendation algorithms. The commonly used metrics are the mean squared error and root mean squared error, and also there exists others such as precision an recall.

My question is after I developed a recommendation system, and after evaluating using those metrics, how can I consider that my system provides good quality recommendations, or in others words, what should be the tresholds or MSE or RMSE values to be considered a good recommendation system.

Thanks. Any suggestions are welcomed.

"Good", I think, is based on the state of the art at the moment.  So I would look at respected models from industry leaders and use their  reported accuracies as a base line for what is "good": since it comes down to what is possible.

There are numerous papers on evaluating recommender systems. See, for example:

Evaluating collaborative filtering recommender systems
Evaluating Recommendation Systems
Evaluating Recommender Behavior For New Users
Evaluating recommender systems from the user’s perspective: survey of the state of the art
Looking for “Good” Recommendations: A Comparative Evaluation of Recommender Systems


How to implement handmade features in a Keras Sequential model? I wanted to know how to implement my handmade features in my Keras model.
Below I have some code where the embedding_matrix is the Stanford Glove NLP dataset.

def BidLstm(maxlen, max_features, embed_size, embedding_matrix):
   inp = Input(shape=(maxlen, ))
   x = Embedding(max_features, embed_size, weights=[embedding_matrix],
                 trainable=False)(inp)
   x = Bidirectional(LSTM(300, return_sequences=True, dropout=0.25,
                          recurrent_dropout=0.25))(x)
   x = Attention(maxlen)(x)
   x = Dense(256, activation="relu")(x)
   x = Dropout(0.25)(x)
   x = Dense(6, activation="sigmoid")(x)
   model = Model(inputs=inp, outputs=x)

I want to implement some extra features (preferably numpy array of features) into my keras model.

Any suggestions?

Very interesting  question , I am  wondering if there  are nough skilled  data science  entusiast   that  could   answer this question , because I am  interested in simmilar stuff as well.
Congratulations  for this  question  man


Why do we move in the negative direction of the gradient in Gradient Descent? It is said that backpropagation, with Gradient Descent, seeks to minimize a cost function using the formula:
$$ W_{new} = W_{old} - learningRate \cdot \frac{\partial E}{\partial W} $$
My question is, if the derivate indicates in which direction the function (the graph of the error with respect to the weights) is decreasing, then why subtract from an already negative gradient?
Why not allow the current direction of the gradient (negative lets say) to be the driving factor for updating the weights:
$$ W_{new} = W_{old} + learningRate \cdot (-gradient) $$

Consider a simple example where the cost function to be a parabola $y=x^2$ which is convex(ideal case) with a one global minima at $x=0$
Here your $y$ is the independent variable and $x$ is the dependent variable, analogus to the weights of model that you are trying to learn.
This is how it would look like.

Let's apply gradient descent to this particular cost function(parabola) to find it's minima.
From calculus it is clear that $dy/dx = 2*x$. So that means that the gradients are positive in the $1^{st}$ quadrant and negative in the $2^{nd}$. That means for every positive small step in x that we take, we move away from origin in the $1^{st}$ quadrant and move towards the origin in the $2^{nd}$ quadrant(step is still positive).
In the update rule of gradient descent the '-' negative sign basically negates the gradient and hence always moves towards the local minima.

$1^{st}$ quadrant -> gradient is positive, but if you use this as it is you move away from origin or minima. So, the negative sign helps here.
$2^{nd}$ quadrant -> gradient is negative, but if you use this as it is you move away from origin or minima(addition of two negative values). So, the negative sign helps here.

Here is a small python code to make things clearer-
import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-4, 4, 200)
y = x**2

plt.xlabel('x')
plt.ylabel('y = x^2')
plt.plot(x, y)

# learning rate
lr = 0.1
np.random.seed(20)
x_start = np.random.normal(0, 2, 1)
dy_dx_old = 2 * x_start
dy_dx_new = 0

tolerance = 1e-2
# stop once the value has converged
while abs(dy_dx_new - dy_dx_old) > tolerance:
    dy_dx_old = dy_dx_new
    x_start = x_start - lr * dy_dx_old
    dy_dx_new = 2 * x_start
    plt.scatter(x_start, x_start**2)
    plt.pause(0.5)
plt.show()

Computing the gradient gives you the direction where the function increases the most. Consider f: x -> x^2 , the gradient in x=1 is 2, if you want to minimize the function you need to go in the direction of -2, same with x=-1 as the gradient is -2.
And as gradients are usually vectors, I don't know what a positive or negative gradient would be if gradient is something like (-1, 1).
https://builtin.com/data-science/gradient-descent


When to use which multiple testing correction? There are a large number multiple testing p-value correction methods. e.g.:
   bonferroni : one-step correction
   sidak : one-step correction
   holm-sidak : step down method using Sidak adjustments
   holm : step-down method using Bonferroni adjustments
   simes-hochberg : step-up method (independent)
   hommel : closed method based on Simes tests (non-negative)
   fdr_bh : Benjamini/Hochberg (non-negative)
   fdr_by : Benjamini/Yekutieli (negative)
   fdr_tsbh : two stage fdr correction (non-negative)
   fdr_tsbky : two stage fdr correction (non-negative)

(based on https://www.statsmodels.org/dev/generated/statsmodels.stats.multitest.multipletests.html)
I have found a lot of pages that explain the methods individually (and why corrections are needed) but I have not found an overview of when to use which method e.g. a comparison table or even better a decision flow diagram as it exists for machine learning methods.
Any ideas? How do I decide which multiple testing correction I should apply?

The Bonferroni and Holm methods lead to the same FWER and disjunctive power when analysing multiple primary outcomes. This is because both methods adjust the smallest p-value in the same way. Similarly, the Hochberg and Hommel methods lead to same FWER and disjunctive power when two primary outcomes are analysed and differences between these methods arise when analysing three or more outcomes
You can read about in this article Link


Is there any machine learning algorithm that can solve this problem? I have a data set of 100000 samples with binary output.

I would like to study the impact of Col_A (a continuous feature) on the output result.

Col_A has values from 0 to 7000000 and when I add this feature to my classifier it gives bad accuracy. I have tried z-score and scalar number but it doesn't change a thing.

I would like to try another method to solve this problem.

I made some plot visualization to my data and I found that there is some range in which the majority of the outputs are negatives. ie. when col_A is ranging between (0 - 200), between (2500 - 2800) and between (5200-5400) the majority of outputs are negatives.

I would like to create a new feature to specify which category my Col_A is in, instead of using Col_A.

PS: I can't use if--else rules because I don't know where to delimit these ranges, I have just analyzed the previous values from my plot. but I need to have a dynamic method for getting those categories.

Is there any type of clustering that can solve this problem?

Does neural network help for this case?

You want a supervised approach. Clustering will not care about your target variable and perform arbitrary splits that don't help.

Likely a decision tree can be helpful here, if you use a good implementation that can split the same feature multiple times (to break it into intervals). There are other approaches you could try, such as piecewise regression etc.

I would like to study the impact of Col_A (a continual feature) on the output result.
  Blockquote

I would say this can be measured statistically and there are a lot of ways through which you can study the impact of Col_A on the output. A simple approach would be to create a model which would randomly guess the output for each input and calculate its log loss. Now build another model only with Col_A features along with the input and calculate the log loss of this model. Compare log losses of both the models. If the log loss is significantly reduced in the latter model, this means Col_A has some valuable information. This can be thought like univariate analysis


How to build a classification pipeline that will pass to another model? Not sure if the title explained it, but I am trying to build a pipeline where it's like a decision tree, but also not. Say for example, I had a picture. The model classified the picture, but now I want to send it to another CNN. How could I create something like this? Sorry if it's not detailed, still figuring it out!

You can use Microsoft Azure, to make two different pipelines. If I am not wrong, I understand that you want to compare the two methods, using a pipeline, therefore Microsoft Azure ML Studio is an easy and quick method to test your ideas.


Regular Expressions in Word I've used optical character recognition (OCR) on a historic directory, and am trying to clean up the text with Microsoft Word. Specifically, I need some help writing a Regular Expression to combine two lines together. For example something that is

John Smith, 87 Bank

Bldg

should actually be

John Smith, 87 Bank Bldg

I've tried several approaches, but haven't been successful at all. Can anyone help me with this?

I have a solution that might not be very standardized but will suffice your need. Copy all of your data into any advanced text editor like, Notepad++ or Sublime Text. Next, Use CTRL+H to toggle the find and replace feature. Find : '\n' and Replace with ''.

simple... use advanced Text Editor
like...
Sublime text ,Notepad  ,Vim or Atom..

paste all Bunch of data on any Editior and just remove '\n and replaced it by "" ..

use FIND AND REPLACE function

Notepad   ,sublime text,vim and atom

use Ctrl   H

find box type "\n" and in replace box ""

done :) ;)

tnx


Discarding non-english words in column I have some non-english words/sentences in my data.
I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the brand names, company names, like NLTK etc. I need some solid solution for the purpose.
Here's what I tried:
def removeNonEnglishWordsFunct(x):
    words = set(nltk.corpus.words.words())
    filteredSentence = " ".join(w for w in nltk.wordpunct_tokenize(x) \
                                if w.lower() in words or not w.isalpha())
    return filteredSentence

string = "NLTK testing man Apple Confiz Burj Al Arab Copacabana Palace Ｗは比較的新しくてきれいなのですが Ｓｈｅｒａｔｏｎ hotelは時々 ＮＹらしい小さくて清潔感のない部屋"

res = removeNonEnglishWordsFunct(string)
Output: testing man Apple Al Palace

Expected output: NLTK testing man Apple Confiz Burj Al Arab Copacabana Palace Ｓｈｅｒａｔｏｎ hotel
```

To Tokenise, clean up symbols (i.e. Normalise), etc. just use one of the widely used NLP libraries, they should be able to do most of the work for you.
Examples include:

NTLK
Spacy
SparkNLP
.. and many more. Perhaps look up some articles comparing their strengths and weaknesses on Google to decide what's best with your project.

As for the detecting English words, that might be slightly trickier, but you can find answers to this already from a bit of Googling. E.g. https://intellipaat.com/community/5638/removing-non-english-words-from-text-using-python
Might also be worth posting some code, output examples and what you're intending to do down the line (e.g. training a neural network?) so that other's can provide further help.
All the best,
Kelvin

To do this, simply create a column with the language of the review and filter non-English reviews. To detect languages, I'd recommend using langdetect.
This would like something like this:
import pandas as pd

def is_english(text):
    // Add language detection code here
    return True // or False

cleaned_df = df[is_english(df["review”])]


ReLU vs sigmoid in mnist example PLEASE NOTE: I am not trying to improve on the following example. I know you can get over 99% accuracy. The whole code is in the question. When I tried this simple code I get around 95% accuracy, if I simply change the activation function from sigmoid to relu, it drops to less than 50%. Is there a theoretical reason why this happens?

I have found the following example online:

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.utils import np_utils

(X_train, Y_train), (X_test, Y_test) = mnist.load_data()

X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)

Y_train = np_utils.to_categorical(Y_train, classes)
Y_test = np_utils.to_categorical(Y_test, classes)

batch_size = 100
epochs = 15

model = Sequential()
model.add(Dense(100, input_dim=784))
model.add(Activation('sigmoid'))
model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer='sgd')

model.fit(X_train, Y_train, batch_size=batch_size, epochs=epochs, verbose=1)

score = model.evaluate(X_test, Y_test, verbose=1)
print('Test accuracy:', score[1])

This gives about 95% accuracy, but if I change the sigmoid with the ReLU, I get less than 50% accuracy. Why is that?

I took your exact code, replaced
model.add(Activation('sigmoid'))

by
model.add(Activation('relu'))

and indeed I experienced the same problem than you: only 55% accuracy, which is bad...
Solution: I rescaled the input image values from [0, 255] to [0,1] and it worked: 93% accuracy with ReLU! (inspired from here):
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.utils import np_utils

(X_train, Y_train), (X_test, Y_test) = mnist.load_data()

X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)
X_train = X_train.astype('float32') / 255
X_test = X_test.astype('float32') / 255

Y_train = np_utils.to_categorical(Y_train, 10)
Y_test = np_utils.to_categorical(Y_test, 10)

batch_size = 100
epochs = 15

model = Sequential()
model.add(Dense(100, input_dim=784))
model.add(Activation('relu'))
model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer='sgd')

model.fit(X_train, Y_train, batch_size=batch_size, epochs=epochs, verbose=1)

score = model.evaluate(X_test, Y_test, verbose=1)
print('Test accuracy:', score[1])

Output:

Test accuracy: 0.934

Potential explanation: when using an input in [0, 255], then when doing the weighted sum for the layer $L$: $z = a^{(L-1)} w^{(L)} + b^{(L)}$, the value $z$ will often be big too. If $z$ is often big (or even if it's often > 0), let's say around 100, than $ReLU(z) = z$, and we totally lose the "non-linear" aspect of this activation function! Said in another way: if the input is in [0, 255], then $z$ is often far from 0, and we totally avoid the place where "interesting non-linear things" are going on (around 0 the ReLU function is non linear and looks like __/)... Now when the input is in [0,1], then the weighted sum $z$ can often be close to 0: maybe it sometimes goes below 0 (since the weights are randomly-initialized on [-1, 1], it's possible!), sometimes higher than 0, etc. Then more neuron activation/deactivation is happening... This could be a potential explanation of why it works better with input in [0, 1].

Because with MNIST, you are trying to predict based on probabilities.

The sigmoid function squishes the $x$ value between $0$ and $1$. This helps to pick the most probable digit that matches the label.

The ReLU function doesn't squish anything. If the $x$ value is less than $0$, the the output is $0$. If its more than $0$, the answer is the $x$ value itself. No probabilities are being created.

Honestly, I'm suprised you got anything more than 10% when you plug it in.


DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20 I was watching Machine Learning A- Z from SuperDataScience but when I was doing below code sample:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

dataset = pd.read_csv('Data.csv')
X = dataset.iloc[:, :-1].values

from sklearn.impute import  SimpleImputer
imputer = SimpleImputer(missing_values=np.nan, strategy='mean')
imputer = imputer.fit(X[:, 1:3])
X[:, 1:3]= imputer.transform(X[:,1:3])

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder_X = LabelEncoder()
X[:, 0] = labelencoder_X.fit_transform(X[:, 0])
onehotencoder = OneHotEncoder(categorical_features =[0])
X = onehotencoder.fit_transform(X).toarray()

I got this warning message:

  /usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/_encoders.py:363: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values.
  If you want the future behaviour and silence this warning, you can specify "categories='auto'".
  In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly.
    warnings.warn(msg, FutureWarning)
  /usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/_encoders.py:385: DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20 and will be removed in 0.22. You can use the ColumnTransformer instead.
    "use the ColumnTransformer instead.", DeprecationWarning)
  And this below message also

  /usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/_encoders.py:363: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values.
  If you want the future behaviour and silence this warning, you can specify "categories='auto'".
  In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly.
    warnings.warn(msg, FutureWarning)

  /usr/local/lib/python3.5/dist-packages/sklearn/preprocessing/_encoders.py:385: DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20 and will be removed in 0.22. You can use the ColumnTransformer instead.
    "use the ColumnTransformer instead.", DeprecationWarning)
  I was reading ColumnTransfer in sklearn website library I didn't understand how to fix these error messages

SampleFile:Data.csv

You can do this to get rid of the deprecation messages

from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

ct = ColumnTransformer(
    [('one_hot_encoder', OneHotEncoder(), [0])],    # The column numbers to be transformed (here is [0] but can be [0, 1, 3])
    remainder='passthrough'                         # Leave the rest of the columns untouched
)

x = np.array(ct.fit_transform(x), dtype=np.float)

In OneHotEncoder, use the parameter handle_unknown, it should look something like this, and now onehotencoder is auto on the dataset X, so you can remove the categorical_features or instead keep auto, removing it will solve the error:

onehotencoder = OneHotEncoder(categorical_features=[i], handle_unknown='ignore')

onehotencoder = OneHotEncoder(handle_unknown='ignore')

Thanks.


Classification problem with many images per instance I am working in the following kind of classification problem: I have to classify every instance as class A or class B using many images of the instance. That is, every training example has not one image (which is the usual thing in image classification), but many images, and the number of images for every training instance is not fixed. That is, instance 1 can have 3 images, and according to these images we have to classify it as A or B, and instance 2 can have instead 5 images.

As any machine learning problem, I am provided with many labelled images and I have to build a classifier.

Although ideas are also welcome, I am looking for a documented way to attack this kind of problem (Kaggles, papers or books, mainly).

My main idea was the following: train a model $f$ that given one image gives a probability of that image being of class A. Then, for every training instance, evaluate $f$ in every image of the instance and compute statistics (aggregate) of the distribution of these probabilities, as the mean, median, maximum and minimum. Then, train a model $g$ that has as inputs these aggregates and use the composition of $f$, aggregates and $g$ as the final model. This idea is a bit simple so I am looking for something better.

This might be a hack, but, have you considered repeating some images to make every instance contain a fix number of images? Let's say you have images A, B, C, D, E, F, G, H, and I. Your dataset might look like this:

A , B
C
D, E, F
G
H, I

Make a function that will randomly select one of the image in an instance and then use that as placeholder. From our example, we now modify our dataset into:

A, B, A
C, C, C
D, E, F
G, G, G
H, I, I

Then, from here, you can use a convolutional neural network with 3 inputs. Here is an example of a CNN with multiple inputs: Multi-Input Convolutional Neural Network for Flower Grading - images of same flowers taken at different angles.

Also, you can shuffle the images in each instance to produce more training samples.

Use logistic regression. It accounts for the uncertainty in image labeling; new images will be classified as A or B with a probability associated with it.


What is an example of normalization (making the row unit norm)? I am learning about standardization and normalization concepts for feature engineering.
Standardization is done for example using z-score where based on the mean and std deviation we re-calculate the values so that the mean is 0 and std deviation is 1. This is a column wise operation.
Where as Normalization is performed row wise (to make the entire row unit norm) – thus making it good for calculating things like cosine similarity. I am looking for an example of what it means to make the row unit norm.

A very simple example is the use of relative frequency instead of absolute frequency, for instance in text classification: if the features represent the raw absolute frequency of every word, then large documents have higher values than small documents. In this case the learning algorithm won't be able to distinguish between a word used 10 times in a small 50 words document (so this word is important for the document) vs. the same word used 10 times but in a large 1000 words document (not very important for the document).
Using relative frequency instead (i.e. dividing by the size of the document) solves this problem. It's an example of row-wise normalization, there are many others.


Neural net only works for small datasets I have a neural network that attempts to solve some regression problem.

The network works fine when the dataset has a small number of training examples, lets say 20. It overfits of course, but the training loss decreases to 0 which is what I want right now.

However, when I take my entire dataset, which has 5000 training examples in it, the loss plateaus at around 375. It won't decrease to zero, no matter how many hidden layers and neurons and activations I add. I tried everything.

What does this mean? Why does this happen and how can I fix it? Is it plausible that it's simply not possible to train a model that fits the training data?

It could mean, that your data is too complex to be learned by the model using a certain number of observations (what is number of features do you have?). On the plot training loss is almost zero for small dataset, but you should rely on the validation loss, not on training one. And validation loss is lower for bigger number of samples, then learning process performes predictable normal

I guess you need to work on data pre-processing more. You need to explore your data:

Check for outliers in every column, you can use z-score for that.
If you have missing data, then you need to see how are you imputing the missing values.
If you have some categorical variables, you need to see how you can encode them like use one-hot encoding or label encoding or binary encoding, target mean encoding etc..

Try to do feature selection and ranking and study the correlations between them using heatmaps or pearson coefficient or if you have categorical variables that may have some sort of ordinal ranking among them you can go for spearman coefficient.

Because one thing is clear your dataset is too complex for the neural net to make out any pattern or establish any kind of linear or non linear relationship amongst the features in your manifold of data created in some high dimensional plane. So you need features that are in some way related to each other and try to reduce the number of redundant features.

One thing that you can do is use RFECV with any gradient boosted tree algorithm and get a baseline of how many features you actually need, and after that try scaling your data. If it is not normally distributed try normalization or otherwise you can use Min max scaler or standard scaler and see what works best for you. Scaling your data properly is very important while using neural networks because if not done properly neural nets tend to become biased and go kind of awry.

You also should try better resampling strategies or increasing you k-folds might help.

If you can give some explanation about your dataset that might help others to answer you queries in a better way. Like:

What are you doing?
What are you predicting?
What is your problem statement?


Best python library for neural networks I'm using Neural Networks to solve different Machine learning problems. I'm using Python and pybrain but this library is almost discontinued. Are there other good alternatives in Python?

UPDATE: the landscape has changed quite a bit since I answered this question in July '14, and some new players have entered the space.  In particular, I would recommend checking out:

TensorFlow
Blocks
Lasagne
Keras
Deepy
Nolearn
NeuPy

They each have their strengths and weaknesses, so give them all a go and see which best suits your use case.  Although I would have recommended using PyLearn2 a year ago, the community is no longer active so I would recommend looking elsewhere.  My original response to the answer is included below but is largely irrelevant at this point.

PyLearn2 is generally considered the library of choice for neural networks and deep learning in python.  It's designed for easy scientific experimentation rather than ease of use, so the learning curve is rather steep, but if you take your time and follow the tutorials I think you'll be happy with the functionality it provides.  Everything from standard Multilayer Perceptrons to Restricted Boltzmann Machines to Convolutional Nets to Autoencoders is provided.  There's great GPU support and everything is built on top of Theano, so performance is typically quite good.  The source for PyLearn2 is available on github.

Be aware that PyLearn2 has the opposite problem of PyBrain at the moment -- rather than being abandoned, PyLearn2 is under active development and is subject to frequent changes.

If you are interested in performance comparisons, Soumith Chintala maintains a set of convnet benchmarks that cover several of the already mentioned python frameworks (TensorFlow, chainer, neon, Theano):

https://github.com/soumith/convnet-benchmarks


How do you use KS-test in a data science report? I'm writing a data science report, I want to find an exist distribution to fit the sample. I got a good looking result  , but when I use KS-test to test the model, I got a low p-value,1.2e-4, definitely I should reject the model.
I mean, whatever what distribution/model you use to fit the sample, you cannot expect to have a perfect result, especially working with huge amount of data. So what does KS-test do in a data science report? Does it means only if we got high p-value in KS-test then the model is correct?

In your case, the null hypothesis $H_0$ is that your sample follows your the distribution that your model has learned. The alternative hypothesis $H_1$ is that it follows some other distribution. Assuming you have fixed your significance level $\alpha$ to be $0.05$ (the most common choice for $\alpha$, but up to you if you want to go lower), getting a p-value lower than that means you should reject the null hypothesis.
The p-value can be interpreted as the probability of a type I error, in other words a false positive: the probability that you reject the null hypothesis when it is in fact true. In your case, rejecting the hypothesis means stating that there is statistically significant evidence that the distribution your model has learned is not the underlying distribution of the sample. So yes, you would like as large a p-value as possible.
You are using a Kolmogorov-Smirnov test to compare your sample to a reference distribution, in this case, so it's a one-sample KS test. The way I would put it is that getting a high p-value means that: "it is highly unlikely that your model has learned a wrong distribution". In other words, it is highly likely it has learned a pretty good approximation of the underlying distribution. However, nothing is certain when doing statistical hypothesis testing!
I'm not sure what you're showing on your plots though, since there doesn't seem to be an empirical cumulative distribution function on them (lines look smooth).

The p-value is interpreted as the probability of a type I error. In other words a false positive: the probability that you reject the null hypothesis when it is in fact true.
You are invoking a Kolmogorov-Smirnov test.
" when I use KS-test to test the model, I got a low p-value,1.2e-4, definitely I should reject the model."  Answer  - Your low p-value does not indicate that the observed distribution does not fit expected distribution. p value simply indicates the chance for comiting
type - 1 error which is quite low in your case. The low value of p i.e. alpha implies that your model predicts very well.  The null hypothesis of no difference between two distributions (observed and predicted) is accepted. In nutshell, the test confirms validity of your model.


Seq2Seq Model training: Encoder vs. Decoder Can someone point me to an article which explains how the model training is done in Seq2Seq? I know "Teacher Forcing" is used but what I found so far hasn't been detailed enough. What I am most confused about is where the training happens? The back-propagation goes back to the encoder?

Any insight here would be really appreciated. Thanks!

You can check medium page, https://link.medium.com/iolQKIWawX.
You can also get more detailed pages on medium itself, shared link includes the references of those.


Python: ggplot geom_histogram() binwidth or bin parameter missing? I am unable to change number of bins when using geom_histogram() function. There is nothing in the documention, except some examples where binwidth=X is used (which is how R:ggplot2 uses it). However, this parameter seems to have no effect on the actual output.

Version: ggplot 0.10.4, Python 3.5 64bit

So for example:

import matplotlib.pyplot as plt
plt.figure()
dataframe['column'].plot.hist(bins=30)
plt.show()

works as expected and draws 30 bins. Meanwhile:

from ggplot import *
gg = ggplot(dataframe, aes(x='column')) + geom_histogram(binwidth=300)
print(gg)

does not change number of bins (I understand that binwidth sets interval for one bin), and have tried multiple values and/or datasets

bins=10 parameter is actually added in latest version (ggplot 0.11) https://github.com/yhat/ggplot/issues/513


How to count the number of missing values in each row in Pandas dataframe? How can I get the number of missing value in each row in Pandas dataframe.
I would like to split dataframe to different dataframes which have same number of missing values in each row.

Any suggestion?

When using pandas, try to avoid performing operations in a loop, including apply, map, applymap etc. That's slow!

A DataFrame object has two axes: “axis 0” and “axis 1”. “axis 0” represents rows and “axis 1” represents columns.

If you want to count the missing values in each column, try:

df.isnull().sum() as default or df.isnull().sum(axis=0)

On the other hand, you can count in each row (which is your question) by:

df.isnull().sum(axis=1)

It's roughly 10 times faster than Jan van der Vegt's solution(BTW he counts valid values, rather than missing values):

In [18]: %timeit -n 1000 df.apply(lambda x: x.count(), axis=1)
1000 loops, best of 3: 3.31 ms per loop

In [19]: %timeit -n 1000 df.isnull().sum(axis=1)
1000 loops, best of 3: 329 µs per loop

If you want count of missing values:

np.logical_not(df.isnull()).sum()


Equivalent of numeric encoding when rows can contain multiple values If we have a column like:

    Name
0  Alice
1    Bob
2   Dave

then, after numeric encoding, it becomes:

    Name
0  0
1  1
2  2

What if, however, we have a column like this:

              Names
0        Alice, Bob
1  Alice, Bob, Dave
2              Dave

One way to encode it would be like this:

   Alice  Bob  Dave
0      1    1     0
1      1    1     1
2      0    0     1

However, this creates a lot of extra columns. Is there a way to encode such a column in a way that doesn't end up with loads of extra columns, just like using numeric encoding instead of one-hot-encoding prevents too many columns from appearing in a column like the first one I showed?

If you're using Python, here's some code to reproduce my DataFrame:

import pandas as pd
df = pd.DataFrame({'Names': ['Alice, Bob', 'Alice, Bob, Dave', 'Dave']})

Edit: the ultimate purpose of this is to then pass this through a tree-based classifier.

The binary encoding seems to be the most natural, capturing exactly the present relationships and not introducing nonexistent relationships.  The combinations of names then are thought of as vertices of a hypercube; naturally, if you collapse this down to one dimension, you will be introducing nonexistent relationships.  Indeed, this is what happens with label encoding: Bob isn't really any closer to Alice than Dave is, but your label encoding makes that appear to be the case.

I've usually been happy enough with just the binary encoding, so I don't have much to back up the following, but some ideas:

For your case, you could still project the hypercube to one dimension; there are any number of ways to do it, and every one will add some additional information that shouldn't be there.  You could experiment and try to find one that seems most suitable for your purposes.

That should sound reminiscent of word embedding, and suggests another method: use a neural network to generate "entity embeddings", a projection from your hypercube to some smaller-dimensional space.  In this way, you uncover combinations of persons that are useful for your problem (maybe you end up with a feature that's "at least one of Alice and Dave", etc.).  You could also do something more transparent, and do some feature engineering by hand to pick out interesting combinations and binning to keep the number of features manageable.  For this you could consider target encoding each combination.  (Indeed, that's a very easy solution by itself, though target encoding can easily lead to overfitting if you have very many people or some rare combinations of people.)

Finally, you might be able to split a row containing (Alice, Dave) into two, one with Alice and the other with Dave.  You can now label-encode (or whatever else), and will need either a model that understands multiple rows per sample point, or a postprocessing method to deal with combining rows.

First, do the OneHotEncoder thing with your Name column which will give you three columns as you have stated above, Namely Alice, Bob, and Dave.

We have this Table after doing OneHotEncoding :

    Alice  Bob  Dave
0   1      1    0
1   1      1    1
2   0      0    1

Now you can call apply function on these columns; pass axis=1 to use the apply function row-wise, then convert the dtype to str and then join them as follows (astype function converts the input into the give data-type):

df['AliceBobDave']= df[df.columns[0:]].apply(lambda x: ','.join(x.dropna().astype(int).astype(str)),axis=1)

The Above will give you output :

     Alice  Bob  Dave  AliceBobDave
0    1      1    0     1,1,0
1    1      1    1     1,1,1
2    0      0    1     0,0,1

Now run this command below to remove the columns not needed in the final dataframe. For ex. in this case we remove Alice, Bob, and Dave using the drop function in pandas:

df = df.drop(['Alice','Bob','Dave'], axis = 1)

After executing this command if we print df using print(df), we get:

    AliceBobDave
0   1,1,0
1   1,1,1
2   0,0,1

This is your desired output.


Text-Classification-Problem, what is the right approach? I'm planing to write a classification program that is able to classify unknown text in around 10 different categories and if none of them fits it would be nice to know that. It is also possible that more then one category is right.

My predefined categories are:

c1 = "politics"
c2 = "biology"
c3 = "food"
...

I'm thinking about the right approach in how to represent my training-data or what kind of classification is the right one. The first challenge is about finding the right features. If I only have text (250 words each) what method would you recommend to find the right features? My first approach is to remove all stop-words and use the POS-Tagger (Stanford NLP POS-Tagger) to find nouns, adjective etc. I count them an use all frequently appeared words as features.

e.g. politics, I've around 2.000 text-entities. With the mentioned POS-Tagger I found:

law:           841
capitalism:    412
president:     397
democracy:     1007
executive:     112
...

Would it be right to use only that as features? The trainings-set would then look like:

Training set for politics:
feature law         numeric
feature capitalism  numeric
feature president   numeric
feature democracy   numeric
feature executive   numeric
class politics,all_others

sample data:
politics,5,7,1,9,3
politics,14,4,6,7,9
politics,9,9,9,4,2,1
politics,5,8,0,7,6
...
all_others,0,2,4,1,0
all_others,0,0,1,1,1
all_others,7,4,0,0,0
...

Would this be a right approach for binary-classification? Or how would I define my sets? Or is multi-class classification the right approach? Then it would look like:

Training set for politics:
feature law         numeric
feature capitalism  numeric
feature president   numeric
feature democracy   numeric
feature executive   numeric
feature genetics    numeric
feature muscle      numeric
feature blood       numeric
feature burger      numeric
feature salad       numeric
feature cooking     numeric
class politics,biology,food

sample data:
politics,5,7,1,9,3,0,0,2,1,0,1
politics,14,4,6,7,9,0,0,0,0,0,1
politics,9,9,9,4,2,1,1,1,1,0,3
politics,5,8,0,7,6,2,2,0,1,0,1
...
biology,0,2,4,1,0,4,19,5,0,2,2
biology,0,0,1,1,1,12,9,9,2,1,1
biology,7,4,0,0,0,10,10,3,0,0,7
...

What would you say?

I think perhaps the first thing to decide that will help clarify some of your other questions is whether you want to perform binary classification or multi-class classification. If you're interested in classifying each instance in your dataset into more than one class, then this brings up a set of new concerns regarding setting up your data set, the experiments you want to run, and how you plan to evaluate your classifier(s). My hunch is that you could formulate your task as a binary one where you train and test one classifier for each class you want to predict, and simply set up the data matrix so that there are two classes to predict - (1) the one you're interested in classifying and (2) everything else.

In that case, instead of your training set looking like this (where each row is a document and columns 1-3 contain features for that document, and the class column is the class to be predicted):

1           2           3           class
feature1    feature2    feature3    politics
feature1    feature2    feature3    law
feature1    feature2    feature3    president
feature1    feature2    feature3    politics

it would look like the following in the case where you're interested in detecting the politics class against everything else:

1           2           3           class
feature1    feature2    feature3    politics
feature1    feature2    feature3    non-politics
feature1    feature2    feature3    non-politics
feature1    feature2    feature3    politics

You would need to do this process for each class you're interested in predicting, and then train and test one classifier per class and evaluate each classifier according to your chosen metrics (usually accuracy, precision, or recall or some variation thereof).

As far as choosing features, this requires quite a bit of thinking. Features can be highly dependent on the type of text you're trying to classify, so be sure to explore your dataset and get a sense for how people are writing in each domain. Qualitative investigation isn't enough to decide once and for all what are good features, but it is a good way to get ideas. Also, look into TF-IDF weighting of terms instead of just using their frequency within each instance of your dataset. This will help you pick up on (a) terms that are prevalent within a document (and possibly a target class) and (b) terms that distinguish a given document from other documents. I hope this helps a little.

The following great article by Sebastian Raschka on Bayesian approach to text classification should be very helpful for your task. I also highly recommend his excellent blog on data science topics, as an additional general reference.
You may also check this educational report on text classification. It might provide you with some additional ideas.


Why perceptron does not converge on data not linearly separable This is how I understand the perceptron algorithm.

The perceptron loss function is the hinge loss $\ell(w,x,y) = \max(0, -yw\cdot x)$.
Suppose the data set is $D = \{(x_1,y_1),\dots,(x_n,y_n)\}$ with $y_i \in \{-1,1\}$.
The perceptron criterion is
$$
E(w) = \sum_i \ell(w, x_i, y_i) = \sum_i \max(0, -y_iw\cdot x).
$$
The perceptron algorithm tries to minimize $E(w)$ using gradient descent.
The gradient of $E(w)$ with respect to $w$ is
$$
\nabla E(w) = \sum_i \left\{
\begin{array}{ll}
\mathbf{0} & \text{ if } \ell(w,x_i,y_i) = 0\\
-y_ix_i & \text{ otherwise }
\end{array}\right..
$$
So the perceptron updates the $w$ according to
$$
w^{(t+1)} = w^{(t)} - \eta \nabla E(w^{(t)}) = w^{(t)} + \eta \sum_{i\in M(t)} y_ix_i
$$
where $M(t)$ is the set of points misclassified by $w^{(t)}$. Since we can define another equivalent sequence of weight vectors $v^{(t)} = (1/\eta)w^{(t)}$, we can just take $\eta = 1$.

I know that $w^{(t)}$ will converge to a seperating hyperplane if one exists and $E(w)$ will converge to $0$.

My question is about when the data set is not linearly separable. In that case, does $E(w)$ converge to some minimum?

If $E(w)$ attains some minimum, its gradient there has to be $\mathbf{0}$. One possibility is that all the terms in the sum of $\nabla E$ are 0, which means that the dataset is linearly seperable and this is contradiction. The other possibility is that the sum of $-y_ix_i$ over the misclassified points is somehow $\mathbf{0}$.

The behavior appears to actually depend on the learning rate $\eta$; a smaller $\eta$ affects which points are misclassified in the next iteration, which affects the weight update more than just by the simple scaling you alluded to.
With appropriately small learning rates though, it seems you are guaranteed convergence to some local minimum, if you avoid certain degenerate situations that would give e.g. a constant loss surface.  Exactly what you describe happens at these minima: the loss to the misclassified points of either class equal each other.
I put together a short demonstration in this colab notebook (github link).  Below are some animations of the evolution of the decision line during the gradient descent, starting at the top with a large learning rate and decreasing it from there.  In this situation, there should be two local minima, the two "tangents" to the "blobs" (possibly pulled in a little, misclassifying a couple of extra points if it can decrease the loss on the central points a little more than the cost for the newly misclassified points).

Let $\{(x_j, y_j)\}_{j\in J}$ be the set of points misclassified by $w$ (the ones with non-zero terms in the sum of $\nabla E$). Under the hypothesis that $-\sum_{j\in J} y_j x_j = 0$,

$$\sum_{j\in J} y_j w \cdot x_j = w \cdot \left(\sum_{j\in J} y_j x_j\right) = 0,$$

therefore the data are linearly separable.


How duplicated items can be deleted from dataframe in pandas I have the following data:

userid itemid
  1       1
  1       1
  1       3
  1       4
  2       1
  2       2
  2       3

How can I drop duplicated items (not the index) to make the following DataFrame:

userid itemid
  1       1
  1       3
  1       4
  2       1
  2       2
  2       3

Can someone help me?

import pandas as pd
df = pd.DataFrame({'userid':[1,1,1,1, 2,2,2],
                   'itemid':[1,1,3,4, 1,2,3] })

The best way would be to use drop_duplicates(). If you have a larger DataFrame and only want those two columns checked, set subset equal to the combined columns you want checked.

df = df.drop_duplicates()

or

df = df.drop_duplicates(subset=['userid', 'itemid'])

Using the following code:

df = df.drop_duplicates(['userid','itemid'])


Adding more layers decreases accuracy I have my ANN trained on MNIST dataset. Hidden layer has 128 neurons and input layer has 784 neurons. This gave me an accuracy of 94%. However when I added one more layer with 64 neurons in each then the accuracy significantly reduced to 35%. What could be the reason behind this.

Edit : Activation function : sigmoid. 521 epochs.

The reason is that by adding more layers, you've added more trainable parameter to your model. You have to train it more. You should consider that MNIST data set is a very easy-to-learn dataset. You can have to layers with much less number of neurons in each layer. Try $10$ neurons for each to facilitate the learning process. You can reach to $100%$ accuracy.

The problem in your case (as I thought previously) is the sigmoid activation function. It suffers from many problems. Out of that your performance decrease is likely due to two reasons:

Vanishing Gradient
High learning rate

NOTE: The link provided for 'Vanishing Gradient' explains beautifully why increasing layers make your network more susceptible to saturation of learning.

The vanishing gradient problem makes sure your Neural Neyt is trapped in a non optimal solution. While the high learning rate ensures that you get trapped in the non optimal solution. In short the high learning rate after a few oscillations will push your network to saturation.

Solution:

Best solution is to use the ReLu activation function, with maybe the last layer as sigmoid.
Use an adaptive optimizer like AdaGrad, Adam or RMSProp.
Decrease the learning rate to $10^-6$ to $10^-7$ but to compensate increase the number of epochs to $10^6$ to $10^7$.


Different Test Set and Training Set Distribution I am working on a data science competition for which the distribution of my test set is different from the training set. I want to subsample observations from training set which closely resembles test set.

How can I do this?

Great question, this is what is known in Machine Learning paradigm as either
  "Covariate Shift", or "Model Drift" or "Nonstationarity" and so on.

One of the critical assumption one would make to build a machine learning model for future prediction is that unseen data (test) comes from the same distribution as training data! However, in reality this rather simple assumption breaks easily and upcoming data (its distribution) changes over time for many reasons. For those who may not be familiar with this very important problem, I encourage looking here or post!

To me, your question falls into the same category. Although I do not have the perfect solution (an implementation to offer), but I think you may look:

This blog post gives you a simple way to handle the subsampling of training data with code provided in Python!
Check this research paper. They propose to solve the problem by reweighting the training data so that the distribution of training is closer to the distribution of test using Kullback-Leibler Importance Estimation Procedure base on "Kullback-Leibler divergence" theorem. I do not know if they provide an implementation or it can be implemented easily, but I think it might worth digging as it sounds a professional way to deal the distribution mismatch.

QUICK update (a good solution): I found a Python implementation of KLIEP algorithm of that research paper (last point) to find those weights. It rather seems easy to use! Basically it resamples the training by putting weights (via the KLIEP algorithm) so that the assumption of having a similar distribution of train and test holds true as much as possible.

There is a good package in python (scikit learn)

http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html

You can  subsample your observations from training set using this package.


Best approach for this unsupervised clustering problem with categorical data? I'm a software engineer new to Machine Learning. I've read about basic non-supervised techniques like k-means and hierarchical clustering and now I'm trying to put them into practice with a basic problem.

Say I've got a lot of rows of data with each row looking something like this:

"employeesRange": "11-50",
"category": "Lobbying",
"categoryGroup": "Polication Action",
"sector": "Industrials",
"subCategory": "Direct",
"tags": [ "Progressive", "Libertarian", "PAC" ]

I want to analyze this data and look for patterns, i.e. clusters of information that frequently appear together. For example maybe it's a common pattern to find smaller groups of lobbyists 11-50 people for "Progressive", "Libertarian" causes but "Presidential" or "Local Politics" groups tend to skew larger.

Or maybe there's another link to be found between categories and sector, that kind of thing.

This seems a little harder then the examples I've read about because some of my data is a tag cloud (the "Tags" field) which is unstructured can contain multiple entries and the more structured fields of categories and employees range.

It seems a good place to start is using hierarchical clustering (since I don't know k a-priori) maybe following the recommendations here:

https://stackoverflow.com/questions/23943391/how-to-cluster-users-based-on-tags

I have a few questions about this. First, is this a good approach? And is Mahout the best tool for this? Any obvious ways to simplify? And finally, how can I combine the tag clustering approach with the other more structured data in the rows?

K-means is a reasonable approach and a sensible way to understand the data.
I've never used mahout, but I would use R or Python for this sort of analysis because of the nice libraries available to quickly implement K-means.
The clustering approach with the tags is fairly straightforward. You can essentially encode this using an indicator variable (also known as a binary encoding). You can set this variable/feature to 1 if the tag appeared in the list of tags and 0 otherwise. Then you only need to allocate space for the total number of tags that exist. If you have a large set of tags, you can limit them by taking tags with at least some frequency or some other "sensible" way.
You can choose $k$ in a number of ways. Typically people choose K arbitrarily because they want, e.g., 10 groups to segment their customers or data into. In the simulation I've provided, it'll give you a lame way to optimize for K using an incremental improvement.

I've made a notebook with a simulation walking you through how to "tokenize" your tags and represent them with a binary/one-hot encoding. It's worth noting that this tokenization ignores the order of the tags, which may be okay for your use case.

It's also worth noting that K-means certainly isn't the only way to measure the similarity of your data but I think it's a nice intuitive start.

Again, for choosing $k$ the approach outlined in the notebook is exhaustive, since it starts from 1 and goes until each observation is a cluster. This means you'll have to run K-means $n$ times, which is silly in practice to do but useful from a learning perspective here. In general, this isn't ideal because it's expensive and typically you don't want to set $k = n$ but this simulation gives nice intuition about what's happening.

In practice, you can just do it in gaps for a large number of clusters (e.g., 5, 10, 15, 20, .., 100) or something like that and choose the one that has the biggest drop-off by eye-balling it. This is a very arbitrary and unsatisfying way to choose $k$, but it seems to work okay for many people.

K-means is neat strategy for these sort of problems. However you could also explore recent methods including topological data analyses to see clusters of features/individuals.


Neural Network stacking layers I was watching Andrew Ng's video on ResNets, and he mentioned that "And in theory, as you make a neural network deeper, it should only do better and better on the training set."
Here is my understanding of neural networks, as the model progresses through the model, the parameters it will learn will become more and more sophisticated, correct? Intuitively, it should be able to recognize/discover more detailed pattern information about the training set. Is my understanding correct?

Then why, in practice, does adding excessive layers to Neural Networks actually harm the performance of the model?
Thanks in advance.

The answer given so far is wrong.
1.) In general, adding more and more layers does not lead to better training accuracy.
2.) The issue is certainly not overfitting, as Andrew Ng is talking only about the training set.
The answer becomes true if you add layers, where each additional layer $l$ is represented by a mapping $f^{(l)}_{w}: \mathbb{R}^{n_{l}} \rightarrow \mathbb{R}^{n_{l}}$ for which weights $w$ exist, such that $f^{(l)}_{w}$ is the identity map. In such a case, you can add arbitrary many layers to an existing neural network architecture and in theory optimizing this new network cannot be worse than without the additional layers (since all additional layers can become the identity map). Potentially these new layers could improve the training accuracy.
In practice this is not observed (see the ResNet paper). The reasons are mainly
1.) The employed solvers do not deliver the global optimal solution.
2.) Adding more layers makes the gradient computation more prone to numerical errors due to the involved chain rule, e.g. have a look here.

Adding more and more layer indeed make the network learn better and better on the training set. However, this causes a problem called "overfit". Overfitting means that your model works extremely well on training set but works poorly on validation set or testing set.

For your last question, the performance of Neural Network models is measured by the ability of the model to predict correctly for unseen data (or future data), which is the accuracy of predicting testing data. In practice, when you make your model deeper, the model will fit more completely to the training data. In this case, you increase the chance of getting an overfitted model. Therefore, the performance of the model (which is the accuracy of predicting testing data) will decrease due to overfitting into training data.

Reference: Overfitting


Is feature selection necessary? I would like to run some machine learning model like random forest, gradient boosting, or SVM on my dataset. There are more than 200 predictor variables in my dataset and my target classes are a binary variable.

Do I need to run feature selection before the model fitting? Does it affect the model performance significantly or is there not much difference if I directly fit the model using all predictor variables?

Feature selection might be consider a stage to avoid.
You have to spend computation time in order to remove features and actually lose data and the methods that you have to do feature selection are not optimal since the problem is NP-Complete.
Using it doesn't sound like an offer that you cannot refuse.

So, what are the benefits of using it?

Many features and low samples/features ratio will introduce noise into your dataset. In such a case your classification algorithm are likely to overfit, and give you a false feeling of good performance.
Reducing the number of features will reduce the running time in the later stages. That in turn will enable you using algorithms of higher complexity, search for more hyper parameters or do more evaluations.
A smaller set of feature is more comprehendible to humans. That will enable you to focus on the main sources of predictability and do more exact feature engineering. If you will have to explain your model to a client, you are better presenting a model with 5 features than a model with 200 features.

Now for your specific case:
I recommend that you'll begin in computing the correlations among the features and the concept. Computing correlations among all features is also informative.
Note that there are many types of useful correlations (e.g., Pearson, Mutual information) and many attributes that might effect them (e.g., sparseness, concept imbalance). Examining them instead of blindly go with a feature selection algorithm might save you plenty of time in the future.

I don't think that you will have a lot of running time problems with your dataset. However, your samples/features ratio isn't too high so you might benefit from feature selection.

Choose a classifier of low complexity(e.g., linear regression, a small decision tree) and use it as a benchmark. Try it on the full data set and on some dataset with a subset of the features. Such a benchmark will guid you in the use of feature selection. You will need such guidance since there are many options (e.g., the number of features to select, the feature selection algorithm) an since the goal is usually the predication and not the feature selection so feedback is at least one step away.

Yes, feature selection is one of the most crucial task for machine learning problems, after performing data wrangling and cleaning.
you can find the functions implementing the feature selection process using XGBOOST feature importance here.

https://github.com/abhisheksharma4194/Machine-learning


imblearn error installing smote I wanna install smote from imblearn package and I got the Following error:

ImportError                               Traceback (most recent call last)
<ipython-input-10-77606507c62c> in <module>()
     66 len(data[data["num"]==0])
     67 #balancing dataset
---> 68 from imblearn.over_sampling import SMOTE
     69 import matplotlib.pyplot as plt
     70 sm = SMOTE(random_state=42)

~\Anaconda3\lib\site-packages\imblearn\__init__.py in <module>()
     33 """
     34
---> 35 from .base import FunctionSampler
     36 from ._version import __version__
     37

~\Anaconda3\lib\site-packages\imblearn\base.py in <module>()
     17 from sklearn.utils import check_X_y
     18
---> 19 from .utils import check_sampling_strategy, check_target_type
     20 from .utils.deprecation import deprecate_parameter
     21

~\Anaconda3\lib\site-packages\imblearn\utils\__init__.py in <module>()
      5 from ._docstring import Substitution
      6
----> 7 from ._validation import check_neighbors_object
      8 from ._validation import check_target_type
      9 from ._validation import check_ratio

~\Anaconda3\lib\site-packages\imblearn\utils\_validation.py in <module>()
     12
     13 from sklearn.base import clone
---> 14 from sklearn.neighbors.base import KNeighborsMixin
     15 from sklearn.neighbors import NearestNeighbors
     16 from sklearn.externals import six

~\Anaconda3\lib\site-packages\sklearn\neighbors\__init__.py in <module>()
      7 from .kd_tree import KDTree
      8 from .dist_metrics import DistanceMetric
----> 9 from .graph import kneighbors_graph, radius_neighbors_graph
     10 from .unsupervised import NearestNeighbors
     11 from .classification import KNeighborsClassifier, RadiusNeighborsClassifier

~\Anaconda3\lib\site-packages\sklearn\neighbors\graph.py in <module>()
      5 # License: BSD 3 clause (C) INRIA, University of Amsterdam
      6
----> 7 from .base import KNeighborsMixin, RadiusNeighborsMixin
      8 from .unsupervised import NearestNeighbors
      9

~\Anaconda3\lib\site-packages\sklearn\neighbors\base.py in <module>()
     20 from .kd_tree import KDTree
     21 from ..base import BaseEstimator
---> 22 from ..metrics import pairwise_distances_chunked
     23 from ..metrics.pairwise import PAIRWISE_DISTANCE_FUNCTIONS
     24 from ..utils import check_X_y, check_array, gen_even_slices

ImportError: cannot import name 'pairwise_distances_chunked' from 'sklearn.metrics' (C:\Users\ASUS\Anaconda3\lib\site-packages\sklearn\metrics\__init__.py)

Try closing and restarting ipython. imblearn requires scikit-learn >= 0.20 and sometimes the runtime loads in an older version of scikit-learn.

If the issue still persists, then reinstall all packages together to make sure they are compatible.

pip install Tensorflow

then,

pip install imblearn

After the installation restart the system, as The imblearn.tensorflow provides utilities to deal with imbalanced dataset in tensorflow, and imblearn uses Tensorflow as backend.


How to store complex tables and structures? I'm wondering about general approaches to storing complex tables and structures. For example, imagine I have a table like this:

               A1 A2 A3 A4 A5 B1 B2 B3 B4 B5 C1 ....Z5
 individuals
 1             .  .  .  .  . .  .  .  .  .
 2             .  .  .  .  . .  .  .  .  .
 3             .  .  .  .  . .  .  .  .  .
 4             .  .  .  .  . .  .  .  .  .
 5             .  .  .  .  . .  .  .  .  .
 ...           .  .  .  .  . .  .  .  .  .

(I'm just showing the row and column names (variables), but the table would be filled, or could be even much larger, with A1 to A999, B1 to B999 colums.)

I can want to perform some calculation on all $Ax$ columns, or on all $x3$ ones. Since it can be difficult to deal with so many columns, it may be better to write this as:

 i  subcase  A B C ... Z
 1  1      .  .  .  .
 1  2      .  .  .  .
 1  3      .  .  .  .
 1  4      .  .  .  .
 1  5      .  .  .  .
 1  6      .  .  .  .
 1  7      .  .  .  .
 1  8      .  .  .  .
 1  9      .  .  .  .
 2  1      .  .  .  .
 2  2      .  .  .  .
 ...       .  .  .  .

I remember seeing that kind of structure several times, sometimes referred to as long format. Later, one can perform arbitrary operations on all rows meeting some condition on "case". This has some advantages, but could be confusing.

One could average or crate the differences of all the $\text{subcase}==1$ rows, only from $\text{individual}==1$, or when $\text{individual}==subcase$,  for example.

But what would be the best way to store the resulting numbers--columnwise or rowise? If I wanted to store the average of all $\text{subcase}==1$ rows, I would need to repeat the number on every row meeting that condition.

Another option would be to keep the wide format structure, but having 2 rows for the headers—one for the letters and a second one for the numbers. I think it could be a natural way, but I've never seen it before.

Are there other structures people can recommend for this problem, or for more complex situations, such as trees?

In third normal form

IndvidualID
CatID
CaseID
Value

Primary Key:  IndvidualID, CatID, CaseID

If you need to constrain CatID and/or CaseID then have a FK

From that structure you can easily loop and create long format or multidimensional array.
It is also space efficient as a null is simply no row.

In sql an avg

select CatID, avg(value)
  from table
 where CaseID = x
 group by CatID

Or you can just loop on those same constraints
Or a tool like LINQ

To be frank, I really donot understand why this is an issue for you. If you want to a generic solution, I will say save it to a csv file. A file with 30,000 rows and 1,000 cols is NOT that big. Even a shell pipe with simple one-liners can do tasks you mentioned. For example, the following one-liner compute the mean of cells located at Ax rows and x3 cols of data.csv (I generate this data matrix of size 30,000x1,000 by filling random integers )

cat data.csv | perl -F',' -lane 'BEGIN{ $avg=0; $num=0; $row=0; } $row++; next if( $row%3!=0);foreach $i (0..$#F){ if($i%26 == 0){ $avg+=$F[$i]; $num++;} }END{ print "avg=" . ($avg/$num); print "num=$num"; }'
avg=0.1998
num=390000

It takes about 3 seconds on my laptop. You may put whatever interested conditions to filter data, collect filtered ones and compute stats.

Since for whatever complicate condition you AT MOST scan each element once, the complexity is thus at most linear. Even for a big data file, this will not be a big issue. By the way, adding extra auxiliary is simply to 'pre-compute' something in advance.


How to measure accuracy of a route prediction I developed a new route prediction algorithm and I am trying to find a metric that informs on how well a prediction was.

This metric is meant to be used offline, meaning that the goal is not to measure the quality of the prediction when it is need in real time. Instead, We are given a set $R=\{r_1,r_2,...r_{|R|}\}$ of routes that occurred in the past and for each $r_i\in R$ we take a small prefix of $r_i$ and provide it as input to the algorithm which in turn outputs a prediction route $p_i$.

Therefore, given the set $R=\{r_1,r_2,...r_{|R|}\}$ and the corresponding predictions set $P=\{p_1,p_2,...p_{|P|}\}$, I want to compare each pair of routes $(p_i,r_i)$ to determine how different $p_i$ is from $r_i$.

Can anyone point me in the right direction?

My first idea was to compute the area comprised between the two routes $p_i$ and $r_i$, but I really do not know how to interpret the output in terms of good or bad result.

Thanks for the attention

You need to define what a "quality route" is in the context of your problem and score based on that. This will likely be something along the lines of the total distance of the route, the expected time to traverse the route, the expected fuel/resource consumption required to complete the route, or some combination of these. You presumably want to predict routes that are as good or better than the routes given in your set R: you need to define what it means for a route to be "better."

Check: A survey of evaluation methods for personal route and destination prediction from mobility traces


Xgboost in R: Dealing with extreme class imbalance My data has an extreme class imbalance - 99.7% is 0's and 0.2% is 1's and almost all the predictor variables (6 out of 7) are categorical. I trained an xgboost classifier after performing an upsampling (using ROSE in R). Evaluated the model on 20% test data and the following is the result,

Recall, as you can see, is not great. Additionally, if I run the model on a completely different test data (meaning not a chunk of the original data), then the recall value drops to exactly 50%. Is this happening because it is overfitting? Can someone please let me know how I can better this model, considering the extreme class imbalance?

I'm a newbie in the field. Any help would be much appreciated. Thanks!

The drop from 64 to 50 doesn't seem like overfitting, if your test set was properly held back from model development.  (And from the confusion matrix it looks like you haven't made the fairly common mistake of upsampling before splitting.)

If the "completely different test data" is not identically distributed to the training data, that can account for the discrepancy.  There aren't universally good ways to account for failure of the iid data assumption; you could try to underfit the model more, on the assumption that there's some underlying large trend that's constant across datasets and the model is finding (and overfitting to) some additional, more localized, trend in the train/validation set.

Try using SMOTE before applying XGboost.

Follow the details from this link: https://www.analyticsvidhya.com/blog/2016/03/practical-guide-deal-imbalanced-classification-problems/


Unsupervised Sentimental Analysis in R How would you evaluate unsupervised sentimental analysis?
I am reading on evaluating sentimental analysis and learning that much of the classification models that are being used, the data has target/label variables. How would you evaluate a survey that is only text with no target/label variables? Right now, I am just giving the responses a sentimental score (positive/negative/neutral) but I am having trouble approaching validation with 1000+ responses. Any ideas/suggestions?

In a truly unsupervised setting, the only possibility is to cluster the documents. Typically this would be done with topic modelling. Then the result could be evaluated by inspecting the words most associated with every topic, and assign a sentiment class to the whole topic/cluster, i.e. all the documents labelled with it.


Data Cleaning Techniques - What to drop? I have been struggling with this problem for a few weeks but it seems to be out of my league. I have a dataset of product sales over the course of 55 weeks. It contains information on the store ID, and the product ID. However, some of the data is missing. Some store IDs do not have continuous data for the 55 weeks, and some item IDs do not have continuous data. Also, combinations of store IDs and item IDs have missing data points. What is the most efficient way to decide what to drop and what to keep for the best results in my model? If the product I am forecasting is continuous that is a good start, but I am not sure what the best combination of stores and items to drop is.

Check the percantege of missing values in each column. If it is high then you should remove this column, otherwise you are able to try to impute the missing values. For the latter you may replace them with the mean (median or mode etc) or try to predict them using the other known values. You should check if these are values are mising at random or not.


Cross validation plus oversampling? I am quite new to machine learning and python as well. I faced an imbalanced dataset and wanna use cross validation and oversamopling like the figure shown.

I realised the Python function below cannot be directly used for this purpose and please advice some codes for this task.

cross_val_score(model, X_train,np.ravel(y_train), cv=n_folds, n_jobs=1, scoring='roc_auc')

Adding to what Himanshu Rai said, you should be careful not to oversample before the StratifiedKFold, once you risk to put the same sample in both the training and testing folds check this, (where you took your image from) and that doesn't really evaluate your model's capacity to data that it never saw. What i did was to use StratifiedKFold, and then oversample or SMOTE (or whatever you want) all the training folds separately, by only looking at that fold's training data. Then use all those folds to validate your models.

Use stratified K-Fold cross validation, it tries to balance the number of positive and negative classses for each fold. Kindly look here for the documentation and examples. If it still doesnt solve your problem of imbalance please look into SMOTE algorithm, here is a scikit learn implementation of it.