How to read contents of a CSV file inside zip file using spark (python) I have multiple zip files containing two types of files(A.csv & B.csv)

/data/jan.zip --> contains A.csv & B.csv
/data/feb.zip --> contains A.csv & B.csv

I want to read the contents of all the A.csv files inside all the zip files using pyspark.

 textFile = sc.textFile("hdfs://<HDFS loc>/data/*.zip")

Can someone tell me how to get the contents of A.csv files into an RDD?

You need to ensure the package spark-csv is loaded; e.g., by invoking the spark-shell with the flag --packages com.databricks:spark-csv_2.11:1.4.0. After that you can use sc.textFile as you did, or sqlContext.read.format("csv").load. You might need to use csv.gz instead of just zip; I don't know, I haven't tried.

This website would help you out. In the topic called Writing a Spark Application, they've described reading file contents from a zip folder. Cheers!


Why spectral clustering results in disjointed cluster? I'm working on a project where I have to dynamically cluster the position of objects with respect to one coordinate. So I'm essentially dealing with subsequent frames and each frame represents a one-dimensional dataset. The intuition behind clustering is to form clusters out of points that are in similar distance to other points within the cluster and can be naturally connected. I use spectral clustering due to its ability to cluster points by their connectedness and not the absolute location and I set rbf kernel due to its non-linear transofrmation of distance. However, in some frames the algorithm results in unnatural assignments. One example is

import numpy as np
from sklearn.cluster import SpectralClustering

X = np.array([[51.08354988], [57.10594997], [70.51259995], [76.74425011],
[61.24844971], [89.00615082], [98.55859985], [61.26575031], [88.35105019],
[87.40859985]])

clustering = SpectralClustering(n_clusters = 4, random_state  = 42,
                                gamma = 5 / (X.max() - X.min()))
clustering.fit(X)

and the result of clustering is presented in a form of swarm plot below, so only x coordinate matters here (each color represents a cluster and labels are array indices):

What I cannot understand is why points marked as red are clustered together as the similarity between points {4, 7} and {5, 8, 9} should be really low. My first thought was that maybe this is caused by unlucky, random initialisation, but I tried with many different random states and the resulting clusters seems to be persistant. So I guess this is connected to the chosen affinity measure (rbf_kernel) and its gamma parameter. As points move with each frame, and distances between them are kind of dependent on their overall range, I tried to set gamma to 5 / (X.max() - X.min()). The intuition behind this was that if the range is bigger, then the distances between points are usually bigger and we should penalise them more to obtain similar values of exp(-gamma * ||x-y||^2) to those obtained within smaller range. But it doesn't seem to work as expected and results in faulty clustering where red cluster is formed out of points divided by green cluster). My expectation would be rather clusters formed as follows: {0, 1, 4, 7}, {2, 3}, {9, 8, 5} and {6} or {0}, {1, 4, 7}, {2, 3}, {5, 6, 8, 9}.

So my question are:

Is affinity choice and its gamma parameter really the problem here?
If so, how can I choose gamma better?
Otherwise, what approach should I consider to deal with faulty assignments with separated points within the same clusters as presented here?
(Side question) Is there any measure/index that would be suitable to automatically compare clusterings with different number of clusters?

@Edit: As it can be observed under this link, those separated clusters occurs for a short period of time, but still, the problem seems to be recurrent.

First, please note that spectral clustering is very sensitive to the affinity kernel. With the standard RBF kernel, my experience is that spectral clustering often isolates outliers (in the spectral space), leaving clusters with numerous observations which can be separated by great distances. This is a major difference with direct k-means: there is no notion of distance anymore, it's just about connectivity.

In the case shown in your example, the results are "strange" because of the low dimensionality: 1D is typically a case where spectral clustering does not provide "logical" results. In addition, you are quite unlucky on several aspects:

The numbers of clusters (4): try it out with different numbers of clusters, and you will find more trivial results
The positions of observations: move #0 to the left and you may find results that make more sense
The low number of observations: #6 is not considered as an outlier, due to its vicinity to the #5, #8, #9 triplet. Indeed, the more points there are in an area, the largest its influence is. This is also how #4 and #7 can be considered part of the (#5, #6, #8, #9) cluster, rather than the (#2, #3) cluster. Try removing #9 and see how the results suddenly make sense.
The chosen kernel (not tried on my side, but it is likely that the results may vary a lot if you increase the gamma value or change the kernel shape)

TL;DR: your case is an example of how spectral clustering can go wrong with low number of observations and dimensions.

I like your approach, but

you said I use spectral clustering due to its ability to cluster points by their affinities and not the exact position which is cool. (checkout Affinityprop also) BUT then you try to cluster with distance/position. If you wanted to make it reproducible try gamma=0 that should only measure distance.

One more note, these affinity based approaches are extremely expensive as data grows (Spectral time complexity is qubic+) I would advise against it unless you are playing with it on small(ish) datasets or you dont have some special approach in mind. For example spectral is designed for graph based problems originally.


generalized likelihood ratio test (GLRT) I am having trouble in understanding the generalized likelihood ratio test (GLRT). Can anyone explain what it is to me, or point me toward an easy-to-understand reference? Is it a supervised or unsupervised method? How GLRT relates to Bayesian approaches?

Likelihood-ratio tests are a mainstay of classical hypothesis testing. The idea is to form the likelihoods of the two hypotheses under consideration, and choose the one with the highest likelihood if their ratio is sufficiently large. Hypotheses come in two flavors: simple, and composite. Simple tests are those for which the hypothesis uniquely defines the distribution; e.g., the mean is that, or the variance is that. If it is not simple, it is composite; e.g., the mean is not equal to something, or the variance is less than something. Generalized likelihood ratio tests apply to composite hypotheses, and the goal is to find the distributions out of all possible options in the hypothesis space that maximize the likelihoods, and consider their ratio.

In Bayesian statistics one considers the posterior probabilities instead of the likehoods, and the corresponding ratio is termed the Bayes factor.

Supervised/unsupervised learning is something else.

I suggest you consult a textbook for worked examples; I think my explanation does not do the subject justice.

The GLRT is normally applied to a 1) composite hypothesis
                                  2) two or more parameter density functions


How to predict tf-Idf vector with Neural Network? I designed a neural network in Torch for prediction of articles. I want to predict, which article will be read by user based on your reading history.

Input to the network is tf-idf vector of article and another information about user.
Output from the network should be tf-idf vector of next article, which user read.

Tf-idf vector is normalized by SoftMax function.

I know the best solution would be, if a output was a representation of classes but there are a lot of articles and by that there will be a lot of classes. So I represent articles with tf-idf vector on the output.

I tried train with DistKLDivCriterion ( Kullback–Leibler divergence ) but it didn't converge. How to design a network for prediction tf-idf vectors? Which criterion should I use?

tmodel = nn.Sequential()
tmodel:add(nn.Sequencer(nn.FastLSTM(537, 300, rho )))
tmodel:add(nn.Sequencer(nn.Dropout(0,5)))
tmodel:add(nn.Sequencer(nn.Linear(300,527)))
tmodel:add(nn.Sequencer(nn.LogSoftMax(527)))

criterion = nn.SequencerCriterion(nn.DistKLDivCriterion())

I hope you do understand, that trying to predict some abstract TF-IDF vector based on read article you are trying to teach a neural network to produce a new article for a given user. It would be a major step in artificial intelligence if achieved, so keep us posted on your progress.

However, if you would think of alternative approaches, I would suggest following steps to make your neural network to work:

You need a classification/clustering model for your articles (which can be based on your TF*IDF, or on some tags assigned to articles). The more classes you could squeeze from your model, the better would your resulting recommender be. On that step, you should be at least able to provide user with articles similar by topic.
You will need at least two-hop data set of user reading current article and hopping to the next one which would be used as a training set for actual predictor. Here, you will train a neural network (or any other model) which will try to predict a next-article class (from step one) based on current article - that would be a recommender system.

Of course, you may, given you have data from (2), train a neural network that would answer the question of "If user reads article A, will the next article contain word B", but that would require you to have the size of output variables of your TF*IDF matrix (for any reasonable dataset that would be at least 20-30k words unless you work with books for toddlers) which makes your model overcomplicated (that would be, actually, a TF*IDF matrix size number of models for every word).

You should not use neural networks for this kind of problem. Neural network works well for classification problems. Here, your problem is somewhat like recommender system problem. You should search some other algorithm for this problem.


Test dataset with categorical variable value not present in train dataset & transformer I want to replace values of a categorical variable ( named 'six' ) by the mean of my target variable ( named 'target' ).

I am fitting a transformer doing just that on a train dataset df and then transform the test dataset df2.

How do I deal with a value appearing solely in the test dataset ?

When fitted on the train dataset the transformer received no mean value of the target variable on that value.

For example :

myarray = np.array([ [ 1 , 1 , 3 , 'v' , 'a' , 'x' , 0 ] ,
                   [ 2 , 2 , 2 , 'v' , 'b' , 'y' , 1 ] ,
                   [ 4 , 5 , 1 , 'w' , 'c' , 'z' , 1 ] ,
                   [ 2 , 1 , 9 , 'w' , 'c' , 'x' , 1 ] ,
                   [ 1 , 0 , 4 , 'w' , 'b' , 'y' , 1 ] ,
                   [ 2 , 2 , 3 , 'v' , 'b' , 'z' , 0 ] ] )

colnames = [ 'one', 'two', 'three' , 'four' , 'five' , 'six' , 'target' ]

df = pd.DataFrame( myarray , columns = colnames )

myarray2 = np.array([ [ 2 , 7 , 3 , 'v' , 'a' , 'x' , 0 ] ,
                      [ 9 , 2 , 2 , 'v' , 'a' , 'y' , 0 ] ,
                      [ 4 , 5 , 1 , 'w' , 'c' , 'k' , 1 ] ]  )

colnames2 = [ 'one', 'two', 'three' , 'four' , 'five' , 'six' , 'target' ]

df2 = pd.DataFrame( myarray2 , columns = colnames2 )

df is my train dataset, df2 my test dataset.

We can see variable 'six' has the k value not existing in the train dataset.

Next :

df[ 'target' ] = df[ 'target' ].astype( 'float64' )

Next ( my homemade transformer ) :

class Cat2TargetMean( BaseEstimator , TransformerMixin ) :

    def __init__( self , col2trans , tgt_col ) :
        self._col2trans = col2trans
        self._tgt_col = tgt_col

    def fit( self, X, y = None ) :
        self._dic_col_p = {}

        for col in self._col2trans :
            p = X.groupby( col ).mean()[ self._tgt_col ]
            self._dic_col_p.update( { col : p.to_dict() } )
        return self

    def transform( self , X , y = None ) :
        for col , dic_p in self._dic_col_p.items() :
            X.replace( { col : dic_p } , inplace = True )
        return X

Then :

 tsf = Cat2TargetMean( [ 'four' , 'five' , 'six' ] , 'target' )

 tsf.fit( df )

 tsf.transform( df )

 tsf.transform( df2 )

Result :

    one two three   four    five    six target
    0   2.0 7.0 3.0 0.333333    0.0 0.5 0.0
    1   9.0 2.0 2.0 0.333333    0.0 1   0.0
    2   4.0 5.0 1.0 1.000000    1.0 k   1.0

'k' value of column 'six' has not been transformed.

I usually replace unseen and NaN values with the global target mean.

There are also already implemented transformers for target encoding that you could use that gives you some options such as smoothing: scikit contrib - target encoder.

You have to fit your homemade transformer on all the data:

tsf.fit( pd.concat([df, df2]))


XGBoost - feature importance just depends on the location of the feature in the data I'm trying to do some feature selection using XGBoost, but the feature importance chart just spits out the features in order of appearance. The feature that is in the first column in the xtrain data is by far most important and then second is second, etc.
It seems like a sign that the model is not working properly as its not really learning anything... any advice on what could be going wrong?
UPDATE:
Correlation Matrix
https://ibb.co/3shDJjD
Model Code:
params = {
  'subsample':0.5,
  'learning_rate': 0.3,
  'max_depth':8,
  'num_parallel_trees' : 20,
  'objective': 'reg:squarederror',
  'verbosity':0,
  }
watchlist = [(train, 'train'), (test, 'val')]
reg = xgb.train(params, train, num_boost_round=5, early_stopping_rounds=5, evals=watchlist)

Results:
[0] train-rmse:0.274535 val-rmse:0.27431
Multiple eval metrics have been passed: 'val-rmse' will be used for early stopping.

Will train until val-rmse hasn't improved in 5 rounds.
[1] train-rmse:0.273472 val-rmse:0.273653
[2] train-rmse:0.272796 val-rmse:0.27341
[3] train-rmse:0.272318 val-rmse:0.27334
[4] train-rmse:0.271943 val-rmse:0.273346
[5] train-rmse:0.271604 val-rmse:0.273374
[6] train-rmse:0.271218 val-rmse:0.273442
[7] train-rmse:0.270927 val-rmse:0.273529
[8] train-rmse:0.270641 val-rmse:0.273561
Stopping. Best iteration:
[3] train-rmse:0.272318 val-rmse:0.27334

Feature importance (note that 0 and 1 are first). If I change the order of the columns in the xtrain, the feature importance will also change and first two columns will always be two most important features.
https://ibb.co/QcHwbNg

You can use this chunk of code to plot the feature importance of your data. It is also possible that your data is columned with decreasing importance.
from xgboost import XGBClassifier
from xgboost import plot_importance
from matplotlib import pyplot
# load data
X =
y =
model = XGBClassifier()
model.fit(X, y)
plot_importance(model)
pyplot.show()


How does deep learning helps in detecting multiple objects in single image? Let's say there are two cars in an image. How can it detect these cars, given that it can detect single car in an image?

Although many solutions in production systems still use a sliding window as described below in this answer, the field of computer vision is moving quickly. Recent advances in this field include R-CNN and YOLO.

Detecting object matches in an image, when you already have an object classifier trained, is usually a matter of brute-force scanning through image patches.

Start with the largest expected patch size. E.g. if your image is 1024 x 768, but always a distance shot of a road maybe you do not expect any car to take up more than 80 x 80 pixels in the image. So you take an 80x80 block of pixels from one corner of the image, and ask your classifier what chance there is a car in that corner. Then take the next patch - perhaps move by 20 pixels.

Repeat for all possible positions, and decide which patches are most likely to contain cars.

Next, take a block size down (maybe 60 x 60, moving 15 pixels at a times) and repeat the same exercise again. Repeat this until you have hit the expected smallest block size for your goal.

Eventually you will have a list of areas within the image, with the probability that each contains a car.

Overlapped blocks both with high probability are most likely the same car, so the logic needs to have thresholds for merging blocks - usually taking the overlapped area with the highest probability score - and declaring there is only one car in that area.

As usual with ML approaches, you will need to experiment with correct meta-params - in this case block sizes, step sizes and rules for merging/splitting areas - in order to get the most accurate results.

Have a simple CNN such as Efficientnet B0 or B1 do the trick for you. Curate your training data such that you have 2 classes: single vs multi. And let the classifier take care of the business for you if localization is not required.


Site for implementing a research papers without code Before a couple of months, I found a web page for implementing papers which are published without code. Someone posts a paper on it and then community competes to implement that paper.

But I don't remember the name of the page and I cannot find it anymore. Any help?

Follow Kristian Kersting, a former professor of mine at University of Bonn who was involved in that event. That was actually a workshop for (if I remember right) NIPS.

UPDATE: well … I found it :D


How can I randomly sample the space of consistent neural networks for given data? Suppose I have a dataset $X$ and target labels $Y$. For a fixed neural network architecture, how can I randomly and uniformly sample from the space of all possible assignments of weights such that the neural network maps $X$ to $Y$?

I am trying to formalize your question before discussing it.
If I understand correctly, you ask for the following:
For $X \subset \mathbb{R}^{n}$ and $Y \subset \mathbb{R}^m$, let $f:X \rightarrow Y$ be a map.
Let $w \in \mathbb{R}^q$ be weights. We consider a neural network $g: \mathbb{R}^{n}\times \mathbb{R}^q \rightarrow \mathbb{R}^m$, and let $g^{(w)}: \mathbb{R}^{n} \rightarrow \mathbb{R}^{m}, x \mapsto g(x,w)$ be the neural network parametrized by $w$.
Now you want to sample from the set $\underline{W}(f,g,X):= \{w \in \mathbb{R}^{q} \mid f = (g^{(w)})_{\mid X} \}$.
However, I think constructing $\underline{W}(f,g,X)$ is very difficult in general.
The following question arises:
Do you already have some $w \in \underline{W}(f,g,X)$ ?
If not, note that $\underline{W}(f,g,X) = \emptyset$ is possible! (its easy to construct an example for that)
Note also that all known universal approximation theorems have some requirements on $f$, and only state that $f$ can be approximated by some neural network. However, for fixed architecture, it might be that there is no $w \in \mathbb{R}^q$ with $f = (g^{(w)})_{\mid X}$ nor that $f$ can be approximated by $(g^{(w)})_{\mid X}$ (e.g. in terms of the uniform-norm).
If you have some $w \in \underline{W}(f,g,X)$, there are certains trivial permutations (e.g. permuting the nodes of a fully-connected layer, or some channels). Apart from that, I am not aware of a full description of $\underline{W}(f,g,X)$. And without further details or constraints, I think its there is no general answer at the moment.
I hope this helps!

It's probably hard to get exactly a uniform distribution on the weights.  One heuristic approximation is to repeat the following many times:
Randomly choose initial weights.  Train the neural network until you get 100% accuracy on the training set.  Save the resulting neural network.
Each neural network is a sample of weights that are consistent with the training set.  Are they uniformly distributed among the set of all such weights?  That seems unlikely.  But they might give an approximation to such a sample.
This might fail, if training never gives you 100% accuracy.  However, research has demonstrated empirically that if you choose a deep neural network architecture with sufficient capacity and you train for long enough, neural nets can memorize the training set and achieve 100% accuracy on the training set [1].  So, if it fails, I'd recommend increasing the size of the network and trying again.  Of course, there are no guarantees -- it can still fail.  It's a heuristic.
[1] Understanding deep learning requires rethinking generalization.  Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals.  arXiv:1611.03530


Speed Date or Dating Conversation Dataset / Corpora I'm working on an LSTM model that uses dating conversation dataset. Any suggestion as to where I can get such data? Any ideas will be much appreciated.

I have a feeling that you are never going to come across this, most of this data is closely guarded. It's no secret that dating sites like OK Cupid use data science on their app, so I think that most of these sites aren't quick to make their data public.

SpeedDate Corpus collected by the third author, and described in Jurafsky et al. (2009)


Is there any advantage in using Particle Swarm Optimization for clustering than K-Means? I have read some paper about using particle swarm optimization. It doesn't look give much different result than K-Means. I tried to use PSO for clustering but the result is pretty much the same with K-Means with some drawbacks like longer execution time and have a lot of different result caused by the random factor.

K-means makes locally the optimal decision. Most of the time (in particular when this objective works reasonably well at all) this works quite well to find a good local optimum. I doubt that using an approach such as PSO gives you any advantage here - the problem is just too simple, and k-means has the speed advantage, while PSO is unlikely to find other optima.
And in particular the better k-means algorithms such as Hamerly, Elkan, ... are so fast, they'll be able to run with hundreds of random restarts in the time needed for just one PSO.

You cannot acutally compare the two, because PSO is an optimization algorithm, and K-means is a clustering algorithm. An optimization algorithm tries to find the best solution to a given problem, such as a clustering problem. If the clustering problem is stated like it is done for K-means, then you should find the same result using PSO, as applying K-means directly.

The problem statement for K-means defines k clusters as k points in space ("cluster centroids"), and all examples that are closest to each of the k points are assigned to that cluster. If the clusters you are trying to find with PSO are defined the same way, you should find the same results. However, if you define the problem differently, like "find k volumes in space with the maximum density and arbitrary shape", then trying to find the solution to this problem with PSO will give you different results.

It's still going to run more slowly than K-means though, because that's the strong point of K-means. It scales really well, because it doesn't involve any pair-wise comparisons between examples.


How to calculate y_score for ROC AUC? how can I calculate the y_score for a roc_auc_score?

I have a classifier, for classes {0,1}, say RandomForestClassifier.

Then, when I apply it to my test data, I will get a list of {0,1}

But roc_auc_score expects y_true and y_score.

As dummy as it might look, after fitting the model, I was making the following:

def find_prob(df):
    def prob(token):
        # the mean is the probability of label {1}
        return df[df.token == token].predicted.mean()
    df['predicted_score'] = df.token.map(prob)
    return df

X_val['predicted'] = model.predict(X_val).astype(int)
X_val = find_prob(tokenize(X_val))

roc_auc_score(y_val, X_val.predicted_score)

where tokenize is only a function which concatenate all the values in a row to a "unique" token string, so that I would be able of finding all instances of a token and check how many were labeled as {0} and how many were labeled as {1}.

Is this the correct approach? Thank you very much.

y_score can either be probability estimates of the positive
    class, confidence values, or non-thresholded measure of decisions

y_score = model.predict_proba(x)[:,1]

AUC = roc_auc_score(y, y_score) # Above 0.5 is good

So, I have found my answer, Now I am going to share it. y_score is the prediction of the x_test, where you can say in my dataset, it is images, in dataset_train. You can get the prediction of the dataset using this -

p = torch.nn.functional.softmax(output, dim=1)
prediction = torch.argmax(p, dim=1)

Where p is the probability and then using argmax, you can get what you want.


Identifying similar data points I have a data set where customers who are non-delinquent with us have defaulted with others. So from the total data set, 30% are those who defaulted with others but 70% are those who are good with us and other others as well.
With the help of 30% customers who defaulted with others (being good/non-delinquent with us), can we identify the similar customers (matching in profile) from the remaining 70% customers who are currently good with us as well as with others, and also to any new customer?
This is required so that we can have an early indication that who are the customers (from remaining 70% & any new addition) most likely turn bad in near future?
I am not sure how to apply any machine learning algorithm (as we just have 1 class defaulted with others) to identify similar customers.

It's generally treated as a binary classification problem, often called credit scoring. You are trying to know whether a specific entity will default with you or not. The fact that the entity has defaulted with another company can be used as feature of your problem. As mentionned by others, there are other approaches, but I honestly think they are in theoretical infency, and can't seem to find any literrature on their practical implementation to credit scoring.

Then you need to build an historical database to learn on, with an associated label. There are two main approaches : either you construct a table based on periodic (yearly) data and set an horizon for the default. It allows you to build a label by observing what happened in the past. Say you have yearly data and consider a 1 year horizon. You will have multiple line for a company : one for each year, say 2010 to 2018, the label will be built on their status the following year, 1 if they defaulted respectively in 2011, ..., 2019, 0 otherwise. Another approach is to consider life-time default, it is better in theory, and probably better match what you want to do, but it is unlikely you will be able to do that, notably because it require complete data and you usually don't keep those that defaulted.

You can learn on that database, but this is a complex process including variable selection, model selection, model training, model validation. Pretty much standard data science, there is a lot of introductory material you should look at, starting with : an introduction to statistical learning, which is free). Then you can go look at ressources oriented towards credit scoring like Credit Scoring and its application, then to supervised learning applied to credit scoring like deep learning for mortgage risk: https://arxiv.org/abs/1607.02470). This will help you measure the risk associated with your current population. More so if you calibrate your model in probability such that the output of your model reflect the probability of default.

However if you want to use that model for making decisions about lending, then the problem will be made more complex, because :

1) Your initial decision will have an impact on the credit situation of said entity. So you actually would need to assess the impact of both legs of your decision on the caracteristic of the entity before putting it in your prediction model.

2) You need to take costs into account. The costs are usually asymetrical and hard to predict. For loans for exemple, not doing the loan will cost you interests on the nominal (it loops to (1) as the % interest you may ask will modify the contract and hence the situation of the entity), but also associated returns (a bit of money for maintaining their account for exemple), while a default would cost you a proportion of the outstanding amount, which is another order of magnitude, and that proportion is often rather difficult to predict (given insurances for exemple).

I am not really aware for advanced literature for points 1) and 2), despite working in the field, I even suspect there is a complete black-out on those topics as they are the one that provide competitive advantages to a given credit lender.

3) You want to avoid social discrimination, mostly for ethic and legal purpose, which is counterproductive to statistical discrimination. There is not much literature on the topic in general, and fewer in credit scoring, except maybe : https://arxiv.org/abs/1610.02413. Depending on your jursidiction this may give you some headaches.

I think clustering is the way to go, but first you have to preprocess data in the proper way.

I would start encoding your observations using an Autoencoder for dimensionality reduction. This DL technique allows you to compress your data, or to take a "summary of it". The compressed representation of your data is often use as a preliminary step in outlier detection tasks, and I think your problem is highly compatible with this approach.
If you don't want to use Deep Learning, you can recur to other dimensionality reduction techniques. PCA might work, for example. (t-SNE is stochastic, and I don't recommend it outside data visualization in lower dimensions.)

Once you have represented your data in a dense, lower dimensional space, there's a number of things you can do.

Employ some measure of distance between observations in this compressed space. If your goal is to understand how different two observations are, that would be the main thing to do. The most common and simple is Euclidean distance, but there's an awful lot of measures you can use (Manhattan, Minkowski, Mahalanobis, ... you name it!).
You can run clustering techniques. The one I use almost any time is, of course, k-Means clustering. With this, you can try to identify groups of more similar observations, this can give you hints on how to spot "good" and "bad" observations.
An alternative is DBSCAN clsutering, that allows you to classify some observations as outliers (the drawback of this model is that you have many hyperparameters to tweak.)
Additionally, you can train a classifier that is fed with the compressed representation of your data. Labeling observations of a dataset as "good" and "bad" could be time consuming, but it might be worthwile if you don't just want to check how different two customers are, but also get a straight classification.


Why is the F-measure preferred for classification tasks? Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is  generally used for (unsupervised) clustering tasks?

The F-measure is the harmonic mean of the precision and recall.

The G-measure (or Fowlkes–Mallows index) is the geometric mean of the precision and recall.

Below is a plot of the different means.

F1 (harmonic) $= 2\cdot\frac{precision\cdot recall}{precision + recall}$

Geometric $= \sqrt{precision\cdot recall}$

Arithmetic $= \frac{precision + recall}{2}$

The reason I ask is that I need to decide which average to use in a NLG task, where I measured BLEU and ROUGE ( where BLEU is equivalent to precision and ROUGE to recall). How should I calculate the mean of these scores?

The Fı-score is preferred to simple classification accuracy in order to counter the problem of imbalanced datasets; if the thing you are looking for occurs only rarely anyway then a naive classifier can always say no and appear to be working very well! A variant on Fı is Fß, where

Fß = (1+ß²) × [ (P × R) ÷ ( (ß² × P) + R ) ]

Vary ß to balance precision and recall. As to the why F or G, I believe it to be empirical - you don't say if you are classifying or clustering in your own application?

If Precision and Recall are similar, F1 is a good single measure to compare different models.

Short and sweet :)


Predict if user will come the next day or not im novice in data science field,
i have a problem statement: to predict which users will come online the next day.

i have two datasets of the user:

1st dataset of the user ( master table ) which has the overal information of the user:

uid totlike totview totshare last_social_activity   last_service_activity   last_active_date    most_active_day   is_active( True if(last_active_day - present date )< 30 )
1   200     500     115      like                   see_news_feed           11/12/2017           Monday             TRUE
2   300     600     2223     share                  see_dashboard           11/13/2017           Monday             TRUE
4   500     237     311      comment                see_news_feed           11/14/2016           Wednesday          FALSE

totlike - total like
totview - total view
totshare- total share

2nd data set of the user ( everyday activity ): Historical Data of user activity with time stamp.

date       uid  total_like  total_view  total_share last_social_activity    last_service_activity
11/4/2016   1     10        20         1            like                        see_news_feed
11/4/2016   2      9        25         2            share                       see_dashboard
11/4/2016   4      1        23         3            comment                     see_news_feed
11/4/2016   3      3        14         4            like                        see_dashboard
11/8/2016   1      4        14         5            comment                     see_news_feed

what is the right way to predict if the user will come tomrrow?

I think that you should stop thinking about modeling and instead start thinking about the meta datasets that you can draw from what you already have. I have also downvoted the previous answer you received because it's making the same mistake.

One of the most consistent mistakes I see in data science (by all levels of scientists) is that they skip the data transform/wrangling stage and just go straight into modeling. That's a MASSIVE mistake and the answer you seek starts well within this stage.

So, I want you to take a step back and think about your data. What do you need to know about a user? What are some of the features of each user? What are the features of when they visited? What day? What time? Was it the morning or the afternoon? How many days in a row was it? What is the gender of the user? And so on and so on.

Your goal is to create a dataset that has one line per user and the same features for all users. From there, you'll be in a better position to make better decisions about modeling and anything else you want to know about your data.

Basically you cannot.

What you may do is to know the total number of users that will connect.

You can to forecast the next connection, by making assumption on the distribution of the time between two connection if a given user. For example, if the rate of connection is constant, then the time between connection obeys to an exponential law $$p(x) = \frac 1 \lambda e^{-\lambda t}, \text{ if } x>0; p(x)=0 \text{ if } x<0$$ where $\lambda$ is the mean time between connection, which is easy to compute.

Curiously enough, the best forecast (the value with a maximum probability) is today ($x=0$). Or tomorrow ($x=1$) if you make your forecast just before mid-night.

You can try a Weilbull model, but I smell you will also get correct but unpractical results.


k-means in R, usage of nstart parameter? I try to use k-means clusters (using SQLserver + R), and it seems that my model is not stable : each time I run the k-means algorithm, it finds different clusters. But if I set nstart (in R k-means function) high enough (10 or more) it becomes stable.

The default value for this parameter is 1 but it seems that setting it to a higher value (25) is recommended (I think I saw somewhere in the documentation).

So I'm a bit confused...
Any advice ?

nstart option attempts multiple initial configurations and reports on the best one. For example, adding nstart=25 will generate 25 initial random centroids and choose the best one for the algorithm. Hope this helps!

You can read more here...

the explanation doesn't appear naywhere in doc. but I think that setting nstart higher let kmeans run nstart(say 50) random initialization of centroids and choose the one that minimize better the cost; therefore you end up with a much more stable clusters becouse of kmeans always choose the better separation for your data


Labels are not given for multiclass classification problem I have probably a weird question.
If you are dealing with a multiclass classification problem, do you always have already determined target output/labels?

I have e.g. a huge data set with a lot of features about different city areas (population, population density, number of services, banks and so on). I want to classify objects (houses, buildings) in these city areas based on these features, whether they are near the city center or not, let's say I want to have 3-5 labels at the end. But I don't know myself yet, how should I determine these labels.
Is there a specific approach to solve this? Had anyone a similar problem? Please advise

P.S. Earlier I calculated the distance between some object (e.g. house) and city center point (based on latitude&longitude). And based on distances I generated labels. But this approach is not universal when we have different cities of different sizes.

P.P.S. Do I have to follow probably unsupervised learning methods? Do clustering and find the clusters. Then analyze the clusters to give meanings to those identified clusters. And then solve the problem as a multiclass classification problem?

Your problem is associated to "unsupervised" learning in machine learning. You do not have a data set that has training data - meaning that data points with correctly specified labels are not known yet.

You can try different approaches to group/label your data set using the given features. You probably have to check by yourself if your model is "auto"-labeling your data correctly.

Clustering (k-Means)
Decision Trees
Auto-Encoding with NNs

More Approaches


K-Nearest Neighbor Classifier Best K Value I created a KNeighborsClassifier for my dataset adjusting the k hyper-parameter (the number of neighbors) in a for loop. The k value was between 1 and 20. The result was the graph below:

How do I interpret this graph? Which would be my best k value?

There's several ways that you can choose your k value for kNN -

You can use the common formula k = sqrt(n) where n is the number of data points in your training set or you can try choosing k where there is a good balance between computation expense vs noise.

Consider your what fits your problem:
Do you care about runtime? The higher the k, the more expensive computationally it is to run.

It looks like around  k= 6-10 you get some diminishing returns - you could set it there to get a good balance between noise and computation cost, but ultimately it is a very arbitrary selection, so pick what suits your use case best.

That is called an Elbow-Curve!  You need to look for the lowest train accuracy, or highest test accuracy, where the curve doesn't bend much more (y-axis), for a given increase in K (x-axis).  In your case, it's around 7, but you could make an argument for 10, I suppose.  As you increase your 'K' the accuracy improves, but only incrementally, and the cost (total compute resources, overall time to finish, etc.) will increase quite significantly.

Here is another example of a similar Elbow-Curve.

In this example, a good K would be around 7.

Here is some sample code for you to test, and further illustrate the point.

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
#%matplotlib inline

df = pd.read_csv("C:\\your_path\\classified_data.csv",index_col=0)
df.head()

df.info()

df.describe()

l=list(df.columns)
l[0:len(l)-2]

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()

scaler.fit(df.drop('TARGET CLASS',axis=1))
scaled_features = scaler.transform(df.drop('TARGET CLASS',axis=1))

df_feat = pd.DataFrame(scaled_features,columns=df.columns[:-1])
df_feat.head()

from sklearn.model_selection import train_test_split
X = df_feat
y = df['TARGET CLASS']
X_train, X_test, y_train, y_test = train_test_split(scaled_features,df['TARGET CLASS'],
                                                    test_size=0.50, random_state=101)

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train,y_train)

KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
           metric_params=None, n_jobs=1, n_neighbors=1, p=2,
           weights='uniform')

pred = knn.predict(X_test)

from sklearn.metrics import classification_report,confusion_matrix
conf_mat=confusion_matrix(y_test,pred)
print(conf_mat)

Result:

              precision    recall  f1-score   support

           0       0.88      0.90      0.89       250
           1       0.90      0.87      0.89       250

    accuracy                           0.89       500
   macro avg       0.89      0.89      0.89       500
weighted avg       0.89      0.89      0.89       500

Continue:

print(classification_report(y_test,pred))

print("Misclassification error rate:",round(np.mean(pred!=y_test),3))

error_rate = []
# Will take some time
for i in range(1,60):

    knn = KNeighborsClassifier(n_neighbors=i)
    knn.fit(X_train,y_train)
    pred_i = knn.predict(X_test)
    error_rate.append(np.mean(pred_i != y_test))

plt.figure(figsize=(10,6))
plt.plot(range(1,60),error_rate,color='blue', linestyle='dashed', marker='o',
         markerfacecolor='red', markersize=8)
plt.title('Error Rate vs. K Value', fontsize=20)
plt.xlabel('K',fontsize=15)
plt.ylabel('Error (misclassification) Rate',fontsize=15)

# sample data avalable here:
# https://www.kaggle.com/shubh247/simple-way-handle-classified-data-using-knn/data


How to maintain accuracy with more classes? I'm working on a CNN model (implemented by keras) that detects landmarks from images. Every landmark (for example Acropolis, Big Ben, Eiffel Tower etc) is as a separate class and is divided into own directory.
The training dataset contains 1600 images, validation 400 images and test 500 images. The training process has 80 iterations.

I noticed that with two or three classes the accuracy is quite good (3 classes => accuracy: 0.8353) but by every next class it's getting to decrease.

Is it so that adding more classes increases features and complexity? What could be the solution for that? Adding more layers or data? I have tried to add more layers but it's rather worsened the result than made it better. Is it because of vanishing gradient? Adding more data is big problem because I just can't find more unique images.
I also changed other config parameters (optimization algorithm, dropout rate etc) but with no result.

The used config parameters are:

NUMBER_OF_CLASSES = 4
BATCH_SIZE = 32
EPOCHS = 80
DROPOUT_RATE = 0.3
LOSS = 'categorical_crossentropy'
OPTIMIZER = 'rmsprop'
METRICS = ['accuracy']
CLASS_MODE = 'categorical'

And constructed model:

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), input_shape=input_shape))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), strides=2))

model.add(Conv2D(64, kernel_size=(3, 3),))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), strides=2))

model.add(Conv2D(128, kernel_size=(3, 3),))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), strides=2))

model.add(Conv2D(256, kernel_size=(3, 3),))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2, 2), strides=2))

model.add(Flatten())

model.add(Dense(256))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Dropout(0.3))

model.add(Dense(256))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Dropout(0.3))

model.add(Dense(NUMBER_OF_CLASSES))
model.add(Activation('softmax'))

--------------
Total params: 721,604
Trainable params: 719,620
Non-trainable params: 1,984

There can be multiple explenations.

But here are some:

Images in train test and valid that are poor for some classes are not representative in train (poperties of these objects are not to be found hence you cant predict on something that you did not atleast partially learn on)- covariate shift

You say adding data is problematic and I understand but just note that more data beats everything else by a mile. Seriously. Hence augment the data, A LOT!. And make sure that for every class properties are representative and to be found in every train test and valid set.

I can start talking about architecture but these are marginal benefits at this point, make sure you have this point right. Without that every discussion about architeture if futile.

I used Keras ImageDataGenerator for data augmentation. Isn't that enough?

train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    vertical_flip=True)

What other methods/tools should I consider?


Clustering analysis for observations with lists as data So I have several samples analyzed for their chemical composition. After data analysis, for each sample, I have a list of compounds found and their corresponding relative abundance. Some compounds are unique but most are actually found in most samples.

I want to do clustering analysis based on these list of compounds. How do I go about this? Specifically how to vectorize my dataset since each observation is actually an array with both numerical (abundance) and categorical (compound label) variables.

K-means would be a fine clustering method for you to start with, though you will have to provide the number of clusters you wish for it to return (not sure if you know that/can figure it out). Otherwise check out DBSCAN.

As for the mix of numerical vs categorical data types, all you will need to do is one-hot encoding on your categorical variables. What that does is that it will take all of the known possibilities for a category and it creates new features out of them. A 1 is assigned if the sample is part of that category, and a 0 if it is not. In this way you can use numerical and categorical at the same time, just make sure to normalize your numerical data!


Text classification- What to do when train and test data have different features I am performing binary text classification. I have to classify a tweet 0 if neutral and 1 if hate speech.

So as general thumb rule i preprocessed my data. create term document frequency and After removing sparse terms i divide my data into train and test.
I train my model using random forest and logistic regression and it worked fine.

set.seed(123)
tweetRand = randomForest(label ~ ., data = train_sparse, importance=TRUE, nTree=500 )
randPridct = predict(tweetRand, newdata = test_sparse)
table(test_sparse$label,randPridct >=0.5)

Its is working fine on test data which divided from raw content.
But when i am running it on a new unseen data it is throwing an exception.

> predicrRand_test=predict(tweetRand, newdata=sparse_4testing)
Error in eval(predvars, data, env) : object 'run' not found

My understanding is that 'run' is a feature present in training but not in unseen test data and during my model training 'run' was included in tdm. In preprocessing of test , run was not in test tdm.

SO how should i deal with these situation. I am new to data science. Please help.

It's perfectly fine to have different features in your training and testing partitions. If the two groups are randomly selected from the same population, one would expect, if there are many possible features, as is frequently the case in text classification, that differences would be observed. To handle this, the standard practice in the field is to train your model on the training partition, and then evaluate on the testing partition with any new features in that grouping being discarded. If you encounter the edge case of a test observation with no features after this procedure, it is common practice to use a heuristic to classify it, bypassing the model entirely (e.g., an observation with no features is classified as neutral).

You are right it is probably due to new features in the unseen test data. This is data science 101, you should have same features in the test/train datasets.

My guess is that it is due to new levels of a categorical variable not present in the train data, but present in the unseen test data and this comes up as an error in the randomforest scoring. To take care of it you should select a randomized sample for test/train from your population and then harmonize the levels of the categorical variable using the techniques given here:

Harmonize levels of a categorical variable in test/train


Predicting credit applications with timeseries I was wondering what the best way is to make a model for predicting credit applications.

I have two tables, which look like this:

   client_nr  yearmonth  total_nr_trx  nr_debit_trx  volume_debit_trx  ... etc.
1          1     201201            94            49           6527529
2          1     201202            85            58           3475518
3          1     201203            94            61          31317405
4          1     201204            85            52          18869967
5          1     201205            93            53           2893105

  client_nr  yearmonth  credit_application  nr_credit_applications
1          1     201201                   0                       0
2          1     201202                   0                       0
3          1     201203                   0                       0
4          1     201204                   1                       1
5          1     201205                   0                       1

The goal is to determine which clients are likely to apply for a credit. So far, I have made sequences of size (months, features) for each client,
using the first table.

My questions:

Would it now be a good idea to create train/test folds based on theclient_nr? Or should I make splits by month?
Should I then select the first $n$ months as features, and create one label from $k$ months after $n$ that indicates whether the client has applied for a credit in those $k$ months? Or is there a better way?
Would it be better to use regression using nr_credit_applications, or classification on credit_application?

Intuitively I think that the model would need some additional features based on the economic context in order to be more accurate, and that would also be a part where the evolution across time really matters.

  Would it now be a good idea to create train/test folds based on theclient_nr? Or should I make splits by month?

If possible you need to have a full time series for a client as an instance, so I'd say splitting on the client number is much better. But select the client numbers randomly from the full set of client ids, because it's possible that the clients number are assigned following a particular order in time.

  Should I then select the first $n$ months as features, and create one label from $k$ months after $n$ that indicates whether the client has applied for a credit in those $k$ months? Or is there a better way?

That depends on the exact goal and on the kind of algorithm being used, but afaik usually with a time series the label is predicted at any given time given the past and current features and the past labels (or past predicted labels).

  Would it be better to use regression using nr_credit_applications, or classification on credit_application?

This is more a matter of convenience for your application or for the algorithm being used, as the accuracy should be very similar (assuming you use the same kind of method of course).


How to handle negative words in word2vec? I am training a big corpus using word2vec and averaging the word vectors to get sentence vectors. What is the best way to address negative words so that negative and positive sentences are away from each other? For e.g.: "After the fix code worked" and "After the fix code did not work" should ideally give sentence vectors which are far from each other. I heard one approach is to look for negative words like "not" and negate the next word vector. Can someone please clarify if that's a good approach or can suggest a better approach?

When you look at the vectors that word2vec generates - negative words may have unique features but can be treated just like positive words. That is to say, as far as the NN is concerned - these are just similar words. You may have to construct "concept vectors" on top of the word vectors to do what you would like to do.

Your parts of speech tagging should automatically mark negating words as ADV. You can then train on these adverbs in conjunction to your verbs as a positive or negative output. Here's an example using spacy:-

import spacy

nlp = spacy.load('en')        # this can take a while
sample_text = u'Do not go.'
parsed_text = nlp(sample_text)
token_text = [token.orth_ for token in parsed_text]
token_pos = [token.pos_ for token in parsed_text]

At this point token_text will be a list of your words and token_pos will be the POS tagging:-

Do - VERB
not - ADV
go - VERB
. - PUNCT

As you can see, "not" is tagged as ADV here. You can now feed this tagged output (or a better parse tree) into a second network to train for a negative or positive output.

Hope this helps.

You can see this paper Querying Word Embeddings for Similarity and Relatedness.


Logistic or Random Forest? I have data on various standardized tests that certain students took and whether or not they ended up passing a specific class in college. Using the pass/fail as the dependent variable and the different scores on the test for the independent variables, I would like to figure out a metric that best predicts the students success in that class; i.e., Student gets these scores on these tests hence the probability that they pass is so and so.

Logistic regression is my first instinct but I was wondering if there are other, possibly better, ways to approach this problem.

In general, it's never a bad idea to use logistic regression as a first stab at a classification problem. If it doesn't work great, it at least gives you a baseline. Random forest would probably work well here as well. Generally, random forests aren't considered very interpretable, but they're actually pretty interpretable if you're interested in understanding the decision process for a single prediction, which it sounds like would be sufficient for your needs here (i.e., the model can tell you a student is likely to perform a certain way because of their behavior on specific tests). A random forest will probably also handle missing data better, which might be useful to you since I'm guessing not all students took the same tests.

Another option you can try here which handles null data extremely well and would also be simple for inference, is naive bayes.


Incorrectly applying random forest model? I am fairly new to random forest models (and data science in general), and was wondering if I am operationalizing the model I created correctly.

Context: I am creating a random forest model to predict employee voluntary turnover.

Problem: Test set has an accuracy of 97% (AUC .992[this seems too high], Precision: 88%, Recall: 99%) vs train set accuracy of 96%, and yet as we keep getting new voluntary terminations, their probabilities based on the model tend to be less than .1. I.E., indicating that they are not going to leave.

I guess that makes sense considering that as of the time when the model was created, those employees were still within the organization and therefore the model accurately classified them as with the company, but accurately classifying whether an employee is still with the organization or not is of no help, I need to be able to identify those employees that have a higher probability of leaving, which is what my understanding of what the RF model would do (I have done this before with logistic regression models).

Possible Explanations for this that I can think of:

1) Data set doesn't seem too imbalanced:  8054 non terms vs 2158 terms

2) Overfitting? But the test set doesn't drop substantially in accuracy

3) high correlated predictor variables?

Code Attached.

Thanks!

I think you correctly identified the issue: if your model tries to classify whether an employee stays or leaves, by definition every employee "stays" as long as they are an employee in the company.

A possible direction would be to design the response variable as "does the employee leave within a year?" (or any specific period of time). This way you can use the records of past employees who have left and past employees who have stayed across time, i.e. you can have several instances corresponding to the same employee at different times. You could add features such as "had a raise or promotion in the past 2 year" for instance.

This looks like a classic case of class imbalance,

You try oversampling the data.
Since you are targeting an event that is less likely to happen, It is good to optimize on high precision and you can compromise on recall.
Looks like you got lucky there, try using k-fold cross-validation while splitting the dataset into test and train, this looks a little odd because your test accuracy is higher than training accuracy. This might be because of several reasons which can only be diagnosed by looking at the datasets.


Is over fitting okay if test accuracy is high enough? I am trying to build a binary classifier. I have tried deep neural networks with various different structures and parameters and I was not able to get anything better than

Train set accuracy : 0.70102
Test set accuracy : 0.70001

Then I tried machine learning algorithms such as KNN and Decision Trees etc. And I found that Random forest Classifier from Scikit-learn with n_estimators=100 gave me

Train set accuracy : 1.0
Test set accuracy : 0.924068

I tried adjusting other parameters such as max_depth, criterion But the decrease in training set accuracy also caused the test set accuracy to drop. Like

Train set accuracy : 0.82002
Test set accuracy : 0.75222

My question is, is this

Train set accuracy : 1.0
Test set accuracy : 0.924068

acceptable ? Even thought the model is over fitting, the test set accuracy is better.

If you properly isolate your test set such that it doesn't affect training, you should only look at the test set accuracy. Here are some of my remarks:

Having your model being really good on the train set is not a bad thing in itself. On the contrary, if the test accuracy is identical, you want to pick the model with the better train accuracy.
You want to look at the test accuracy. That is your primary concern. So pick the model that provides the best performance on the test set.

Overfitting is not when your train accuracy is really high (or even 100%). It is when your train accuracy is high and your test accuracy is low.
it is not abnormal that your train accuracy is higher than your test accuracy. After all, your model has an advantage with the train set since it's been given the correct answer already.

At the end of the day, training a machine learning model is like studying for a test. You (the model) use learning resources such as books, past exams, flash cards etc. (train set) to perform well on a test/exam (test set). Knowing your learning resources perfectly doesn't mean you are overfitting. You would be overfitting if this is all you knew and couldn't perform well on the exam at all.

The purpose of a model will always be to minimize loss. not increase accuracy. so parameters of any model using any optimizer like adam optimizer(a common optimizer), will try to gain momentum towards parameter values where the loss is least, in other words "minimum deviation".

models can overfit when:

data is small
Train to Test ratio imbalance
model has improper gates or neurons are too rigid.( neurons give high weight age to most recent input and remain locked without considering other inputs).
in DNN's when intermediate weights dont have a forget/reset factor. usually 0.2

but in your case we dont have extreme accuracy like 0.99< . so its safe to say that your model is performing good and is not overfitting. good models do not overfit, they strictly converge to an arbitrary value as 0.924 in your case.


How to detect cardboard boxes using Neural Network I'm trying to train a Neural Network how to detect cardboard boxes along with multiple classes of persons (people).

Although it's easy to detect persons and correctly classifies them, it's incredibly hard to detect cardboard boxes.

The boxes look like this:

My suspicion is that box is too simple of an object, and the neural network has a hard time detecting it because there are too little features to extract from the object.

The division of the dataset looks like this:

personA: 1160
personB: 1651
personC: 2136
person: 1959
box: 2798

Persons are wearing different safety items, based on the items are classified, while detected as whole person, not just the item.

I tried to use:

ssd300_incetpionv2
ssd512_inceptionv2
faster_rcnn_inceptionv2

All of these are detecting and classifying persons much better than boxes. I cannot provide exact mAP (don't have it).

Any ideas?

Thanks.

As you said:

  My suspicion is that box is too simple of an object, and the neural
  network has a hard time detecting it because there are too little
  features to extract from the object.

... and that is exactly the problem with this task. I suggest you to train your Network using a whole lot of image augmentation. If you are using Keras/TensorFlow 2.0, they have built-in functions that do that.

I also suggest you to train a first model, then study accurately the images that your Network thinks are boxes but aren't (the false positives). At that point, you select the "correct" cases, together with the false positive cases, and build a subset of the dataset that you have. That sub-dataset can be useful to train the Network to distinguish between actual boxes and objects that look like boxes.

Once again: use a massive load of image augmentation. That's my main point, and it's what I would do.

You can try using existing Tensorflow models that have been pre-trained on a large datasets such as MS-COCO, Kitti, and Open Images etc. You can then fine tune a particular model that you wish on your dataset.

A list of all available models to use can be found here: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md

A good source on how to set up an object detector : https://www.youtube.com/watch?v=Rgpfk6eYxJA (Don't worry if you are Linux user, the same tutorial can be adapted even in Linux systems)

You can use a tool like labellmg to label your images for training(https://github.com/tzutalin/labelImg)

Hope this helps.


Looking for an algorithm that correctly clusters visually separable clusters I have visualized a dataset in 2D after employing PCA. As 2D visualization shows in figure, there is a good separation between points (A, B). Now, I want to use a metric which can separate these points (between these 2 PC components not in main dataset) too. I mean have separation between these PCA components without visualization. I used some clustering methods but they raise false positives. I mean they miss cluster many points.

Also, as shown in histogram there is a gap between points A,B. Does this help in devising any metric?

I will be so grateful if you can introduce me any method and algorithm to be able to do separation between A and B.

With appropriate parameters, DBSCAN and single linkage hierarchical agglomerative clustering should work very well. Epsilon=0.2 or so.

But why? You know the data, just use a threshold.

If you just want an algorithm to "confirm" your desired outcome then you are using it wrong. Be honest: if you want your result to be "if $F-factor-1 > 1.5 then cluster1 else cluster2", then just say so, instead of attempting to find a clustering algorithm to fit to your desired solution!

Using k-means clustering algorithm on this dataset should work perfectly fine. You just have to pass the (n_samples, 2) matrix where element $(i,j)$ represents the j-th coordinate of sample i in the PCA to any k-means algorithm, and specify that you want 2 clusters, and Euclidean metric.


Does neuron have weight? My understanding is that a connection between two neurons has a weight, but a neuron itself does not have a weight.

If connection c connects neurons A to B, then c has a weight w, but A and B don't have a weight. w determines if A has a strong influence on B or a weak influence on B.

But this Wikipedia article says:
"The connections between artificial neurons are called ‘edges’.
Artificial neurons and edges typically have a weight that adjusts as learning proceeds."

https://en.wikipedia.org/wiki/Artificial_neural_network

The 2nd sentence seems to imply that a neuron has a weight, too.

Does a neuron really have a weight?

If you look at the 2nd equation under "Propagation function"

$p_{j}(t)=\sum _{i=1}^No_{i}(t)w_{ij}+w_{0j}$

The $w_{0j}$ is the constant (bias), so it's reasonable to write that the $w_{ij}$ weights are associated with the connections, and the $w_{0j}$ is a weight associated with the nueron itself.

Additionally if you assume $o_{0}(t)=1$ then you can rewrite the equation above as

$p_{j}(t)=\sum _{i=0}^No_{i}(t)w_{ij}$

(i.e. note the sum is from 0 to N, not 1 to N)

When you do this it no longer really makes sense to make a distinction between a "weight" and a "bias".

Neurons does have a value, which we multiple with the weights to get the activation value for a given neuron.

We generally don't call it as weight of neuron but it means the same.

I'm updating my answer. Let's say we have a input x1,x2 . We want two hidden neuron h1 and h2.

We multiply x1 and x2 with w1 and w2 to get h1:

h1=w1*x1+w2*x2

Similar for h2.

h2=w3*x1+w4*x2

Now h1 and h2 are neurons having some values(that i called as their weight). We use those neurons for next layer value calculation using another set of weights.
Hope that clear things out.


When would one use Manhattan distance as opposed to Euclidean distance? I am trying to look for a good argument on why one would use the Manhattan distance over the Euclidean distance in machine learning.

The closest thing I found to a good argument so far is on this MIT lecture.

At 36:15 you can see on the slides the following statement:

"Typically use Euclidean metric; Manhattan may be appropriate if different dimensions are not comparable."

Shortly after, the professor says that, because the number of legs of a reptile varies from 0 to 4 (whereas the other features are binary, only vary from 0 to 1), the "number of legs" feature will end up having a much higher weight if the Euclidean distance is used. Sure enough, that is indeed right. But one would also have that problem if using the Manhattan distance (only that the problem would be slightly mitigated because we don't square the difference like we do on the Euclidean distance).

A better way to solve the above problem would be to normalize the "number of legs" feature so its value will always be between 0 and 1.

Therefore, since there is a better way to solve the problem, it felt like the argument of using the Manhattan distance in this case lacked a stronger point, at least in my opinion.

Does anyone actually know why and when someone would use Manhattan distance over Euclidean? Can anyone give me an example in which using the Manhattan distance would yield better results?

According to this interesting paper, Manhattan distance (L1 norm) may be preferable to Euclidean distance (L2 norm) for the case of high dimensional data.
The authors of the paper even go a step further and suggest to use Lk norm distances, with a fractional value of k, for very high dimensional data in order to improve the results of distance-based algorithms, like clustering.

Manhattan Distance is the L1 norm form (L1 norm is the sum of the magnitude of vectors in space), while Euclidean Distance is L2 Norm form (The L2 norm calculates the distance of the vector coordinate from the origin of the vector space.


Pound notation in Summation I was going through a paper comparing glove and word2vec. I came across the pound notation shown below. What does it mean when used like this?

The link for paper is here

In the third paragraph of the first section (page 1), they define $\#(w)$, $\#(w; c)$, and $\#(c)$. Quoting such paragraph:

  Word-context pairs are denoted as $(w; c)$, and $\#(w; c)$ counts all observations of $(w; c)$ from the corpus. We use $\#( w ) =  \sum_c \#(w; c)$ and $\#(c) = \sum_w \#(w; c)$ to refer to the count of occurrences of a word (context) in all word-context pairs. Either $\sum_c \#(c)$ or  $\sum_w \#(w)$ may represent the count of all word-context pairs.

So, considering the equation you cited:

$\#(w_i, c_j)$ counts the occurrences of a specific word $w_i$ over the context $c_j$;
and $\frac{\#(c_)}{\sum_w \#(w)}$ accounts for the the probability of having context $c_j$ in the dataset.

That just means count. #(w) is the number of times w occurs in the corpus.


Why are we taking the square root of the gradient in Adagrad? This is how we update weights with Adagrad:
$$w_i = w_i - \frac{lr}{\sqrt{g_i+E}}$$
where, $w_i$ is the $i^{th}$  weight, $lr$ is the learning rate, $g_i$ is the gradient of the $i^{th}$ weight at all the timesteps and $E$ is epsilon to prevent division by zero.
Now why are we taking the square root of $g_i+E$, can't we just remove the root and just divide.
I know removing the root gives bad performance, but why does it gives bad performance on removing the root?

We are taking the square root of $G_i$  because $G_i$ is the sum of squares of the gradient of weight $i$ from iterations $1$ to the current iteration. Since we squared all the gradients of all timesteps, $G_i$ was very large, and so we take its square root. Now you may think that, then why don't we just add the gradients instead of squaring the gradients, then adding. The answer to this question is that we square it to prevent negative gradients, so that negative gradients and positive gradients don't cancel each other out. Now you may think that why don't we take absolute value of the gradient then, the answer to this question is
here
That post was about root mean squared error or mean absolute error.
If you think carefully, these both issues are the same.
So, we take the square and not the absolute value. Now you may think that why don't we  power the gradient to some even power, why only square? Now, I leave this question to the consideration of the reader


Why do internet companies prefer Java/Python for data scientist job? I see a many times in job description for data scientist asking for Python/Java experience and disregard R. Below is a personal email I received from chief data scientist of a company I applied for through linkedin.

  X, Thanks for connecting and expressing interest. You do have good
  Analytics Skills. However,  all our data scientists must have  good
  programming skills in Java/Python as we are a internet/mobile
  organisation and everything we do is online.

While I respect the decision of the chief data scientist, I am unable to get a clear picture as to what are the tasks that Python can do that R cannot do. Can anyone care to elaborate? I am actually keen to learn Python/Java, provided I get a bit more detail.

Edit: I found an interesting discussion on Quora.
Why is Python a language of choice for data scientists?

Edit2: Blog from Udacity on Languages and Libraries for Machine Learning

So you can integrate with the rest of the code base. It seems your company uses a mix of Java and python. What are you going to do if a little corner of the site needs machine learning; pass the data around with a database, or a cache, drop to R, and so on? Why not just do it all in the same language? It's faster, cleaner, and easier to maintain.

Know any online companies that run solely on R? Neither do I...

All that said Java is the last language I'd do data science in.

The tools in Python are just better than R. Ther R community is pretty stagnant while the Python community is evolving really quick. Especially in tools for Data Science.
Also Python works way easier with everything around it. You can easily scrape the web, connect to databases and so on. That makes prototyping really fast.
And if you have a working prototype and care to make it faster or integrate it into the company workflow, it gets usually reimplemented in Java.

R has a few neat tools and visualization but it is not that great to build new stuff in it.


Scaling values for LSTM I have the following time series data set

Each row is a unique Item, and each column shows the amount purchased per day. There are a total of 33 columns.
I'm taking the first 32 columns(leaving out the last column, which will be my target) as my training set, and the last 32 rows (leaving out the first column) as my testing set
X_train = dataset[:, :-1] # taking all columns except the last column

y_train = dataset[:, -1:] # setting the last column to be the target

X_test = dataset[:, 1:] # taking all columns expect the first column

I'm going to feed X_train and y_train in my LSTM model, and use the model to perform prediction on X_test.
Now, I wish to performing minmax scaling on dataset before performing training,  but I have some questions:
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(-1, 1))
scaled_dataset = scaler.fit_transform(dataset)

This has obvious data leakages, because the scaler is fitted with the test values.
I thought of creating 2 scalers, one for the training set, and one for the target
scaler_x = MinMaxScaler(feature_range=(-1, 1))
scaler_y = MinMaxScaler(feature_range=(-1, 1))
scaled_all_data = scaler_x.fit_transform(dataset[:, :-1])
scaled_y = scaler_y.fit_transform(dataset[:, -1:])

I'm not sure if that's the right approach. I've already tried searching for answers, but their situation is not quite like mine, or the questions are unanswered yet.
Any advice on how I should perform value scaling?

Why not scale the values as the very first step? Import your values, scale the values and then do your splits for training/test/etc.


Linear Regression and learning rate Could you please tell me why do we use a learning rate to move into the direction of the derivative to find the minimum? Why is it not good if you simply count it where is it 0?

Learning rate gives the rate of speed where the gradient moves during gradient descent. Setting it too high would make your path instable, too low would make convergence slow. Put it to zero means your model isn't learning anything from the gradients.

Let's take the function Y = x^2. Where is its minimum? It's where dy/dx = 0. Let's find that. dy/dx = 2x. But why 2x? Or simply how do we find a derivative function for any function just that easy? It's because we exactly know the relationship between Y and X. That is Y is always X*X. And same for all functions. Let's come to some data. Do we know the exact relationship between the input X and the target Y? No. Because we can't say 'Y is always "this" relationship with X' and that is actually what we r trying to find out. So what can we do ? Take small baby step in a way that our error is  minimizing. Making big move might lead us to bad place.

So to answer ur question, in a case of Y = x^2, we know the relationship, so we find the minimum by formula not by analysis. But in case where the problem is mapping input data X to target Y, (where we don't know the relationship), we can't use formula to get minimum but analysis. U can use analysis for the former case, but it's little tedious I guess.


How to find similarity of two series over time containing periodic trends? Considering the data is received from a streaming source each second.How to distinguish if both the line graphs 'look' same/different in real time, statically, like the picture given below

Edit:
1.Not sure if there is something as real-time  periodic correlation mechanism then cross-correlation would perhaps be an ideal solution ?
2. Comparing slope of the two Line is the last option I would go with.
3. If statistically there is no way to solve this then I would look at machine learning to solve this

Maybe it is a huge oversimplification, but you can try: What if you do just arithmetic difference between the two signals (or divide one signal to another one) and look at the result?

I would expect that you will have spikes where signals become very different and you can make a threshold to find where they are on time.

be series1 = S and series2 = T

It would come from:

Tn - Sn-1 = Sn - Tn-1, then:

Tn - Tn-1
-------------- = 1
Sn - Sn-1

In this way, favorable cases will give 1 and unfavorable cases will show noise.


Using attributes to classify/cluster user profiles I have a dataset of users purchasing products from a website.

The attributes I have are user id, region(state) of the user, the categories id of product, keywords id of product, keywords id of website, and sales amount spent of the product.

The goal is to use the information of a product and website to identity who the users are, such as "male young gamer" or "stay at home mom".

I attached a sample picture as below:

There are all together 1940 unique categories and 13845 unique keywords for products. For the website, there are 13063 unique keywords. The whole dataset is huge as that's the daily logging data.

I am thinking of clustering, as those are unsupervised, but those id are ordered number having no numeric meaning. Then I don't know how to apply the algorithm. I am also thinking of classification. If I add a column of class based on the sales amount of product purchased. I think clustering is more preferred. I don't know what algorithm I should use for this case as the dimensions of the keywords id could be more than 10000 (each product could have many keywords, so does website). I need to use Spark for this project.

Can anyone help me out with some ideas or suggestions?

Thank you so much!

Right now, I only have time for a very brief answer, but I'll try to expand on it later on.

What you want to do is a clustering, since you want to discover some labels for your data.  (As opposed to a classification, where you would have labels for at least some of the data and you would like to label the rest).

In order to perform a clustering on your users, you need to have them as some kind of points in an abstract space.  Then you will measure distances between points, and say that points that are "near" are "similar", and label them according to their place in that space.

You need to transform your data into something that looks like a user profile, i.e.: a user ID, followed by a vector of numbers that represent the features of this user.  In your case, each feature could be a "category of website" or a "category of product", and the number could be the amount of dollars spent in that feature.  Or a feature could be a combination of web and product, of course.

As an example, let us imagine the user profile with just three features:

dollars spent in "techy" webs,
dollars spent on "fashion" products,
and dollars spent on "aggressive" video games on "family-oriented" webs (who knows).

In order to build those profiles, you need to map the "categories" and "keywords" that you have, which are too plentiful, into the features you think are relevant.  Look into topic modeling or semantic similarity to do so.  Once that map is built, it will state that all dollars spent on webs with keywords "gadget", "electronics", "programming", and X others, should all be aggregated into our first feature; and so on.

Do not be afraid of "imposing" the features!  You will need to refine them and maybe completely change them once you have clustered the users.

Once you have user profiles, proceed to cluster them using k-means or whatever else you think is interesting.  Whatever technique you use, you will be interested in getting the "representative" point for each cluster.  This is usually the geometric "center" of the points in that cluster.

Plot those "representative" points, and also plot how they compare to other clusters.  Using a radar chart is very useful here.  Wherever there is a salient feature (something in the representative that is very marked, and is also very prominent in its comparison to other clusters) is a good candidate to help you label the cluster with some catchy phrase ("nerds", "fashionistas", "aggressive moms" ...).

Remember that a clustering problem is an open problem, so there is no "right" solution!  And I think my answer is quite long already; check also about normalization of the profiles and filtering outliers.

For orientation and exploration, I can recommend WeKa, which is a very nice toolkit for machine learning. It does take a certain input format (.ARFF) so you might need to look into that as well.

As for the keyword dilemma, I recommend performing some feature selection in order to eliminate redundant or non-indicative keywords.


Doesn't over(/under)sampling an imbalanced dataset cause issues? I'm reading a lot about how to use different metrics specifically for imbalanced datasets (e.g. two classes present, but 80% of the data is one class) and how to tackle the issue of imbalanced datasets.
One trick is to oversample, so to take more (or even duplicate some) data belonging to the underrepresented class. I've tried this and did achieve better results (before my models would easily just predict a single class for everything, achieving 80% accuracy lol).
However, I was wondering, will this model work well with real-life data? One of the 'laws' of data science/machine learning is that your training data has to have the same/similar attributes as your live data you're intending to use your model on. However, by oversampling, I create a dataset that's 50% one class and 50%  other, as opposed to the "natural", real-life-data having 80% of one class and 20% of the other.
So I guess the question in short is: Will oversampling my imbalanced dataset of 80/20 class distribution to 50/50 class distribution impact the usability of my model for real-life data? Why?

Yes, the classifier will expect the relative class frequencies in operation to be the same as those in the training set.  This means that if you over-sample the minority class in the training set, the classifier is likely to over-predict that class in operational use.
To see why it is best to consider probabilistic classifiers, where the decision is based on the posterior probability of class membership p(C_i|x), but this can be written using Bayes' rule as
$p(C_i|x) = \frac{p(x|C_i)p(C_i)}{p(x)}\qquad$ where $\qquad p(x) = \sum_j p(x|C_j)p(c_j)$,
so we can see that the decision depends on the prior probabilities of the classes, $p(C_i)$, so if the prior probabilities in the training set are different than those in operation, the operational performance of our classifier will be suboptimal, even if it is optimal for the training set conditions.
Some classifiers have a problem learning from imbalanced datasets, so one solution is to oversample the classes to ameliorate this bias in the classifier.  There are to approaches.  The first is to oversample by just the right amount to overcome this (usually unknown) bias and no more, but that is really difficult.  The other approach is to balance the training set and then post-process the output to compensate for the difference in training set and operational priors.  We take the output of the classifier trained on an oversampled dataset and multiply by the ratio of operational and training set prior probabilities,
$q_o(C_i|x) \propto p_t(x|C_i)p_t(C_i) \times \frac{p_o(C_i)}{p_t(C_i} = p_t(x|C_i)p_o(C_i)$
Quantities with the o subscript relate to operational conditions and those wit the t subscript relate to training set conditions.  I have written this as $q_o(C_i|x)$ as it is an un-normalised probability, but it is straight forward to renormalise them by dividing by the sum of $q_o(C_i|x)$ over all classes.  For some problems it may be better to use cross-validation to chose the correction factor, rather than the theoretical value used here, as it depends on the bias in the classifier due to the imbalance.
So in short, for imbalanced datasets, use a probabilistic classifier and oversample (or reweight) to get a balanced dataset, in order to overcome the bias a classifier may have for imbalanced datasets.  Then post-process the output of the classifier so that it doesn't over-predict the minority class in operation.

Undersampling should mostly not be preferred because it causes a huge amount of data loss. In the end, we are giving so much effort to collect data and it basically does not make sense when we throw them away. The issue is here is that you lose samples where your model could learn new things.
Oversampling usually works better as you observed but the problem is that the synthesized samples might be noisy and might not perfectly reflect real-world situations. So, it also creates some deficiencies. So, to what extent oversampling will work depends on your capabilities of generating synthesized samples that are just like real-world samples. Of course, it is again dependent on the problem's complexity. If your generated samples are not like real-world samples, your model will learn on noisy samples which will probably reduce your accuracy in the wild.


Comparing AUC, logloss and accuracy scores between models I have the following evaluation metrics on the test set, after running 6 models for a binary classification problem:

  accuracy logloss   AUC
1   19%      0.45   0.54
2   67%      0.62   0.67
3   66%      0.63   0.68
4   67%      0.62   0.66
5   63%      0.61   0.66
6   65%      0.68   0.42

I have the following questions:

How can model 1 be the best in terms of logloss (the logloss is the closest to 0) since it performs the worst (in terms of accuracy). What does that mean ?
How come does model 6 have lower AUC score than e.g. model 5, when model 6 has better accuracy. What does that mean ?
Is there a way to say which of these 6 models is the best ?

I will start with your last question as the answer explains the other questions. In general, this is a hard question. That is exactly why there are different metrics available. Depending on your problem and what you want to achieve, you should choose the metric that measures this best.

The differences in metrics can be found all over the internet. Others have explained it much better than I can.


When to use GRU over LSTM? The key difference between a GRU and an LSTM is that a GRU has two gates (reset and update gates) whereas an LSTM has three gates (namely input, output and forget gates).

Why do we make use of GRU when we clearly have more control on the network through the LSTM model (as we have three gates)? In which scenario GRU is preferred over LSTM?

GRU is related to LSTM as both are utilizing different way if gating information to prevent vanishing gradient problem. Here are some pin-points about GRU vs LSTM-

The GRU controls the flow of information like the LSTM unit, but without having to use a memory unit. It just exposes the full hidden content without any control.
GRU is relatively new,  and from my perspective, the performance is on par with LSTM,  but computationally more efficient (less complex structure as pointed out).  So we are seeing it being used more and more.

For a detailed description, you can explore this Research Paper - Arxiv.org. The paper explains all this brilliantly.

Plus, you can also explore these blogs for a better idea-

WildML
Colah - Github

Hope it helps!

GRU is better than LSTM as it is easy to modify and doesn't need memory units, therefore, faster to train than LSTM and give as per performance.


When to use One Hot Encoding vs LabelEncoder vs DictVectorizor? I have been building models with categorical data for a while now and when in this situation I basically default to using scikit-learn's LabelEncoder function to transform this data prior to building a model.

I understand the difference between OHE, LabelEncoder and DictVectorizor in terms of what they are doing to the data, but what is not clear to me is when you might choose to employ one technique over another.

Are there certain algorithms or situations in which one has advantages/disadvantages with respect to the others?

There are some cases where LabelEncoder or DictVectorizor are useful, but these are quite limited in my opinion due to ordinality.

LabelEncoder can turn [dog,cat,dog,mouse,cat] into [1,2,1,3,2], but then the imposed ordinality means that the average of dog and mouse is cat. Still there are algorithms like decision trees and random forests that can work with categorical variables just fine and LabelEncoder can be used to store values using less disk space.

One-Hot-Encoding has the advantage that the result is binary rather than ordinal and that everything sits in an orthogonal vector space.  The disadvantage is that for high cardinality, the feature space can really blow up quickly and you start fighting with the curse of dimensionality. In these cases, I typically employ one-hot-encoding followed by PCA for dimensionality reduction.  I find that the judicious combination of one-hot plus PCA can seldom be beat by other encoding schemes.  PCA finds the linear overlap, so will naturally tend to group similar features into the same feature.

You can also use frequency encoding in which you map values to their frequencies
Example taken from How to Win a Data Science Competition from Coursera,
eg. for titanic dataset:
   encoding = titanic.groupby('Embarked').size()
   encoding = encoding/len(titanic)   // calculates frequency
   titanic['enc'] = titanic.embarked.map('Encoding')

This will preserve info about values distribution and will help both linear and tree based models.
Make sure you don't have multiple categories with same frequencies, you can use rank operation instead or any other relevant operation.


Visualizing decision tree with feature names from scipy.sparse import hstack
X_tr1 = hstack((X_train_cc_ohe, X_train_csc_ohe, X_train_grade_ohe,X_train_price_norm, X_train_tnppp_norm, X_train_essay_bow, X_train_pt_bow)).tocsr()
X_te1 = hstack((X_test_cc_ohe, X_test_csc_ohe, X_test_grade_ohe, X_test_price_norm, X_test_tnppp_norm, X_test_essay_bow, X_test_pt_bow)).tocsr()

X_train_cc_ohe and all are vectorized categorical data, and X_train_pt_bow is bag of words vectorized text data.
Now i applied decision tree classifier on this model, i got this.

i took max_depth as 3 just for visualization purpose.

my question is i want to get feature names in my output instead of index as X2599, X4 etc.
i know i can do it by vect.get_feature_names() as input to export_graphviz, vect is object of CountVectorizer(), since i have already merged this vectorized data using hstack, now how to get feature names in this decision tree.

hstack preserves the order of the columns, so you can piece together the feature names for each of your component arrays.  OneHotEncoder (if that's what you used) and CountVectorizer both support get_feature_names, so concatenating the lists of feature names should be possible.  To give full details would require more details about how each of the arrays was generated.  You might consider using ColumnTransformer in the future, which handles all that concatenation for you and also provides its own get_feature_names.

You can use graphviz instead. and use the following code to view the decision tree with feature names.

import pydotplus
import sklearn.tree as tree
from IPython.display import Image

dt_feature_names = list(X.columns)
dt_target_names = [str(s) for s in Y.unique()]
tree.export_graphviz(dt, out_file='tree.dot',
    feature_names=dt_feature_names, class_names=dt_target_names,
    filled=True)
graph = pydotplus.graph_from_dot_file('tree.dot')
Image(graph.create_png())

This will display feature names with values, gini coefficient, sample, value and class


Recommending college degrees based on high school subjects I have a list of our college students' high school courses.

I want to recommend college degrees to current high school students based on their courses - that is, predict a class based on a vector. For instance someone who studied algebra, calculus and statistics could be recommended software engineering, accountancy or mathematics.

There seem to be a few approaches I could take: market basket analysis, collaborative filtering, clustering or even neural networks.

I can structure my data in a sparse matrix of courses, with each row having a class representing the student's eventual degree, e.g.:

DEGREE       English    Calc    Algebra    Geography    History ... etc
Soft.Eng.    0          1       1          0            0
Comms        1          0       0          1            1
Mech.Eng.    1          1       1          0            0

How should I approach this?

This problem is perfectly suited for a neural network. Your model will have 40 input nodes (this is fine), then you will have some arbitrary hidden layers, you need to tune this, and 20 outputs. After the training process you can even get a probability for each of them. This can be used to rank suggestions for potential students!

How to do this

Load the data to memory

Depending on the means by which your data is stored this step will be different. However, the goal is to go from the raw file source to either Python Numpy array or a Pandas DataFrame. I will assume your data is structured as follows

and is saved as a .csv file.

Let's load our data into an X and Y matrix. We will be encoding the labels for 'degree' as values. Make sure these are all well spelled otherwise a new label will be created for the mispelled ones.

import pandas as pd
import numpy as np

df = pd.read_csv('test1.csv')
df['Degree'] = df['Degree'].astype('category')
df["Degree_encoding"] = df["Degree"].cat.codes

X = np.asarray(df.loc[:, df.columns != 'Degree'])
Y = df['Degree_encoding']

print(X.shape)
print(Y.shape)

  (39, 7) (39,)

Applying a neural network to this dataset

First we will split the data randomly into a training and testing set. This is used to evaluate our model while maintaining that we are not overfitting. Then we identify the number of output classes. Then we reshape the input matrices such that they have a channel in their last dimension, this is how data flows through the model. Then we will categorize our outputs as one-hot encoded vectors.

from sklearn.model_selection import train_test_split
import keras

# Split the data
x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

# The known number of output classes.
num_classes = len(set(df["Degree_encoding"]))
# Input dimensions
shape = X.shape[1::]

# We need to add a channels dimension to our data
# Channels go last for TensorFlow backend in Keras
x_train_reshaped = x_train.reshape((x_train.shape[0],) + shape)
x_test_reshaped = x_test.reshape((x_test.shape[0],) + shape)
input_shape = shape
print(input_shape)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

We then design our model

model = Sequential()
model.add(Dense(32,
                 activation='relu',
                 input_shape=input_shape))
model.add(Dense(128, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

You can use model.summary() to get a description of these layers. Then we are ready to train our model!

epochs = 100
batch_size = 128
model.fit(x_train_reshaped, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test_reshaped, y_test))

On such data, naive bayes (and maybe non-naive Bayes variants) should perform extremely well. Because all your inputs are binary.

It's also incredibly cheap to train, evaluate, and explain.

You can combine it with frequent itemset to make it less naive, but I wouldn't be surprised if that does not improve the accuracy much. You would use FIM to find dependencies worth modeling, and build an optimal Bayes for non-overlapping frequent subsets (e.g. engineering subjects vs. languages). Then combine these partitions assuming independence.