Which algorithm to use to predict the duration of some task I have a data set of 36K processes each with 7 features. Each set represents a task and I know how long it took to complete each of the tasks. I want to build a model which will be able to predict the duration of the future tasks.

I've only learned about Decision Tree (DT) and tried to apply it to my problem. The resulting accuracy score is 0.03. I believe DT is not suitable because the time is continuous and DT is meant for categorization.

Which algorithm is suitable for duration prediction?

My environment: Python with sklearn, if that matters.

What others have said is accurate, that you need to build a regression model of some sort. Depending on the scale of the duration of your task, you will have to model it slightly differently.

In fact, there's an entire class of models that try to predict duration. These are called survival models. Here's a Python library on survival analysis.

But these models are fairly academic. A typical hack is to use Gamma, Poison, or Log-Normal regression which works out nicely because these models predict non-negative values.

You might want to look into scikit's DecisionTreeRegressor() class. A decision tree regressor will predict a real number. The decision tree classifier, by contrast, will predict a discrete class for an observation. A more advanced version would be the RandomForestRegressor() class, which builds a random forest of regression trees. A third option to consider might be a GradientBoostedRegressor().

I would also recommend this flow chart from Scikit that can help you choose an estimator.


Is there a method that is opposite of dimensionality reduction? I am new to the field of machine learning, but have done my share of signal processing. Please let me know if this question has been mislabeled.

I have two dimensional data which is defined by at least three variables, with a highly non-linear model way too complicated to simulate.

I have had varying level of success at extracting the two main components from the data using methods like PCA and ICA (from the python library Scikit-Learn), but it seems these method (or at least, these implementation of the methods) are limited to extracting as many components as there are dimensions in the data, for example, 2 components from a 2D point cloud.

When plotting the data, it is clear to the trained eye that there are three different linear trends, the three color lines show the directions.

When using PCA, the main component is aligned to one of the color lines, and the other is at 90°, as expected. When using ICA, the first component is aligned with the blue line, and the second is somewhere in between the red and green ones. I am looking for a tool which could reproduce all three components in my signal.

EDIT, Additional info: I am here working in a small subset of a bigger phase plane. In this small subset, each input variables produce a linear change on the plane, but the direction and amplitude of this change is non-linear and depends on where exactly on the bigger plane I am working. At some places, two of the variables can be degenerate: they produce change in the same direction. for example, say the model depends on X, Y, and Z. A change in the variable X will produce a variation along the blue line; Y causes a variation along the green line; Z, along the red one.

The short answer is yes.

Essentially you will be performing some sort of feature engineering. This means constructing a series of functions of your data, often:

$\phi_j(x): \mathbb R^p \rightarrow \mathbb R \ ,\ \  j=1,...,K$

Which, strung together, define a transformed data vector $\boldsymbol \phi(x)$ of length $K$.

There are a number of ways, better and worse, of doing this. You may want to look up terms like:

Splines and generalised additive models.
The kernel trick (how to make a model where $K\rightarrow \infty$).
Feature engineering (of the manual variety, e.g. adding an $x^2$ column to your data).
Deep learning, representation learning

As you might guess from such a varied bag of techniques, this is a large area. It goes without saying really but care has to be taken to avoid overfitting.

This paper Representation Learning: A Review and New Perspectives deals with some of the issues around what makes a particular set of features 'good', from a deep learning perspective.

I guess you are looking for features which extract out new features. A feature which best represents the dataset. If that is the case then we call such method "feature extraction".


Is the search space of Hyperparameters Continuous or Discrete? I am looking into hyper-parameter tunning and was curious about whether the search space is considered continuous or discrete?
My understanding of both those cases:
1. Continuous would make it 'easier' to look for hyperparameters combinations that are more fine-tuned to the problem.
2. Discrete would imply that a more brute-force (or random) approach would be helpful.

Subsidiary question: Am I mistaking on my understanding of either of those cases?

Thanks

Continuous means only you have continuous variables. It can be convex or concave. It might not even be differentiable. Gradient descent only applies to differentiable convex problems (or convex approximations of concave problems).

Example:

learning rate of SGD is continuous hyper parameter.
NN optimizer (SGD, Adam, RMSprop) is discrete parameter

From my perspective: right now, rarely we have resources to perform cross validation on a number of deep models. Most of the hyper parameters are tuned using intuition, a bit of math and tuned in isolation.

If you can afford it:

grid search to initialize solutions
if your'e unhappy with the results:

run some hill climber on space, like simulated annealing
run simple GA, there'll be problems because of mixed space (some params are continuous, some are discrete), you'll need custom operators

Yep, you are understanding it right.

The Hyperparameter search is another optimization problem. You have to search along with all the parameters.

If it was continuous you could apply gradient descent for example. But since it is discrete you can´t search the space this way. There are other strategies considered here, random search, grid search, Bayesian or even genetic algorithms to try to find the optimal parameters


Are there any annotators or Named Entity Recognition for license plate numbers? Most vehicle license/number plate extractors I've found involve reading a plate from an image (OCR) but I'm interested in something that could tag instances of license plates in a body of text. Are there any such annotators out there?

There are a lot of pretty decent tools out there for text annotation in general, and given the broad nature of the task you're approaching (license plates are about as general as words), the annotation tools you are looking at should probably come from the more classical tools for annotation.

There was actually a pretty good discussion about annotation tools on this question, which should actually apply to this problem. The most relied-upon thing in annotation right now is probably brat. You can learn more about brat here.

Hope that helps! Let me know if you've got any more questions.

This can be done using regular expressions.

2 letters followed by a number (\d denotes digits) would be

[A-Z]{2} \d*

2 or 3 letters followed by a number is

[A-Z]{2,3} \d*


Meaning of Perceptron optimal weights Im studying the perceptron algorithm.
I know that we can use the weights w as the coefficients of the hyperplane that will separate the vectors that need a classifications

In every web page i read up for a detailed explanation, it's generally said that the perceptron algorithm will find the optimal weights w. But optimal in which sense ?

Media's explanation is true for regression problems. These are problems where you predict a continuous target variable.

Your image shows a classification problem. Here, the target variable takes only two values (typically -1 and 1 in the Perceptron algorithm).
In that case, an optimal solution $w^*$ is a vector of weights that perfectly separates both classes.
If such a solution exists, the Perceptron algorithm will find it. But: If there is one optimal solution, there are usually infinitely many other optimal solutions. You can easily see this in your image: You can move the line a little to the left or to the right, and you can rotate it a little, and it still perfectly separates the classes.

So while the Perceptron algorithm will find an optimal solution if there is one, you cannot know which one it will find. That depends on the random starting parameters.

This is different e.g. for support vector machines. Here, there is either no optimal solution or exactly one optimal solution.

Using perceptron, you specify a cost function, Mean Squared Error for regression tasks or maybe Cross Entropy for classification tasks. The input data are the constants and the weights are the parameters of your learning problem. When you specify the cost function, if you have error, the cost would be non-zero. You use algorithms like gradient descent to decrease the cost value. This is an optimization problem which you try to decrease the value of error. When we say Perceptron finds the optimal point, the reason is that the shape of cost function, e.g. MSE is convex and there is just one optimal point which gradient is zero there and the cost has the least possible value there. If you use neural networks, the cost with respect to its parameters, weights, is not convex and you usually can not find the optimal point.

I suggest you looking here and here for understanding more neural nets optimality.


What is advantage of using the theory of point processes instead of probability distribution functions? I am interested in applications of point processes in machine learning, so I have been studying the theory of point process. However, I cannot see advantage of the theory of point processes compared with the classical probability theory.

For examples, I know the points of a Poisson point process scatters uniformly on a given domain. But, I'm confused what is the difference between Poisson point processes and uniform distribution functions (especially on a bounded domain).

I have similar questions for not only Poisson point processes but other point processes. Take sample points according to a point process. Can we obtain a similar (random) point set via using a probability distribution function?

What is advantage of using the theory of point processes instead of using probability distribution functions in statistics and machine learning?

Thank you so much in advance.

Machine learning is about finding a model for your data and sometime the Point process is a better model than uniform or normal probability distribution.


How does Performance function classify predictions as positive or negative? Package:ROCR I'm performing a logistic regression on my training data. I used the glm function to get the model m. Now using the below codes from this link, I calculated AUC

  $test\$score<-predict(m,type = 'response',test)$
  $pred <- prediction(test\$score,test\$good_bad)$
  $perf <- performance(pred,"tpr","fpr")$

where score is the dependent variable (0 or 1).
To score the tpr (True positive rate) and fpr (False positive rate), you have to classify the predicted probabilities into 1 or 0.
What is the cutoff used for that? how can we change it?

Could not find anything useful in this main documentation as well.

I cant access an R console at the moment to check, but I'm quite certain the cutoff is 0.5: if your glm model does prediction, it first produces real values and then applies the link function on top. To the best of my knowledge, you can't change it inside the glm function, so your best bet is probably to check ROC, find what the optimal threshold is and use that as cutoff.

If you're not sure which label ROCR took as +ve then check str(pred_obj@labels) and the greater one shown is considered +ve; if you want to change that then use label.ordering argument supplying a vector containing -ve and +ve labels while creating prediction object


Regression - random error term When we use a regression algorithm in out dataset it's because we assume that there is a relation between our input data and some quantitative value. This is expressed as :

$y = f(x)+\varepsilon $, where x is an input vector and $\varepsilon$ is the random error term.

Now, this random error term can have any probability distribution ?

$\varepsilon$ is referred as a noise term with 0 mean. The distribution is random in the real world, but you can make assumptions on its distribution.

For example the Gaussian Process machine learning suggests that it follows a Gaussian distribution i.e :  $\varepsilon \sim \mathcal{N}\left(0, \sigma^ 2 \right)$.

The variance $\sigma$ of the latter distribution can be seen as an hyperparameter that we obtaining by maximizing the likelihood function, or by having prior information on the data set.
You can find more information in this book Rasmussen, C. E., and C. K. Williams. "I (2006) Gaussian Processes for Machine Learning." (2006)

In some cases by looking at your a priori data, and estimating the possible error sources, you can a priori expect the type of the error distribution and try to assess this assumption a posteriori (mainly using data-driven method).

No, if it's a random error it has to follow a normal distribution with mean value equal to zero.

This because of the central limit theorem.

If instead that error do not follow a normal distribution, than it is not random: it's a systematic you probably have to consider into the original function f(x).


Newbie question on restricted boltzmann machine I’m quite a newbie to RBMs so I’m trying to understand how do you feed real valued data to it given that all the visible and hidden units are binary?

Here is an answer which might help: https://www.quora.com/What-are-best-approaches-to-modeling-real-value-continuous-input-variables-in-RBM-pretraining


Traditional Predictive Analytics vs Machine Learning Methods What is the difference between traditional predictive analytics done using statistics and its tools and, one using machine learning and deep learning?
 How are we leveraging machine learning and deep learning to make predictive models better?
How to decide the boundary for when to use traditional models and, when to use ML and DL?
We have a lot of efficient statistical tools available Stata, SPSS. They are easier to use. Also, computationally efficient.

Thanks

I am unable to find some satisfactory and convincing answer with strong points on google.

This question is very hard to answer because ML is very new and not as semantically and academically defined as statistics.

However, it helps to see the problem a bit different by focusing on tools and methods as well as goals and use cases.

Per se machine learning uses statistical and mathematical algorithms to solve computational problems and specifically problems of prediction. This is true for traditional predictive analytics and tools as well.

A random forest model is not intrinsically traditional or ML and indeed I can fit it in SPSS as well as in python.

So what if any are the differences:

First, we have to understand that ML comes from a different domain than classical statistical analysis. For many cases, there will be no real difference but a CS student may call it ML while a Sociologist will call it predictive analysis.

So let's look at more specific questions of you:

Why not use STATA and SPSS?

You can! Indeed STATA and SPSS do have ML capabilities themselves and could be used to model modern ML algorithms. However, for many specific use cases in the ML domain, STATA and SPSS lack computational power and the ability to model large data sets efficiently.

Why not use traditional models like linear regression, etc. all the time?

Not every problem and every type of data fit classical methods. Linear regression, for example, cannot cope with sparse data and large instances of NAs, it cannot predict nonlinear relations and collapses really fast if we use a lot of predictors (a simple image can be converted to 786 predictors for example).

Additionally ML methods are just updating classical predictive analytics by implementing them using state of the art technology. ML is different from traditional statistics mainly in usage, where we have stuff like incremental learning and self-optimization, etc.

Also please understand that there is a huge difference between classical ML algorithms like boosted trees, random forest, Naive Bayes, etc. and deep-learning which is really, really different.

It's an interesting question.
According to my understanding, first of all, it will be totally incorrect to say that a statistical approach and machine learning are the same things (even though they might look like).

--Statistics plays a major role in building any machine learning model.
But Machine Learning aims to predict something based on the training set and compare to a test set, whereas statistics help us build a relationship between variables.

--Take the basic example of linear regression, where it seems as if both statistical methods and ML are the same(which is not true). Linear regression is a statistical method, we can train a linear regressor and obtain the same outcome as a statistical regression model aiming to minimize the squared error between data points. But in ML, you will be dividing your dataset into a training dataset and build your model from it and test it on a 'test set' to see how it worked. But statistics just focus on the underlying relationship between the variables. Even though this can be used for predictive analysis you mentioned in the question, but the evaluation will not be on the basis of train vs test results, rather by evaluating the significance and robustness of the model parameters.

--ML can be used for building supervised and unsupervised models where statistical methods can help you forecast continuous variables, regression, and classification. ML uses it for sure but has much more objective to it than just finding a relationship.

I hope the answer gives you something to think about.


Any case studies using Bayesian Networks for system design trades? I am exploring using Bayesian Networks to identify the best parameters within a system design, to improve its performance. I'm trying to find any case studies where this has been used successfully or unsuccessfully.

As an example consider an automatic assisted anti-collision system on a car.  It could use some combination of ultrasonic, radar & IR seekers to sense and measure the closing speed with the car in front.  These all have different costs / reliabilities and perform differently in different weather conditions.  I am interested in understanding how effective Bayesian Networks can be in evaluating the trade space and identifying the best solutions to these types of problems.

As I can see you're searching for a case study. Learning from data is useful because you can learn both the conditional probability distributions (CPDs) and the structure of the network. The best resource I've read is a freely-available Microsoft tech report.

Have a look at the paper, maybe it can help you upto some extend. I'll suggest you to start with Chapter 7 of it.

Hope it helps. Cheers!

There is a paper that was published at Systems&Cybernetics journal.

See how they use Bayesian Networks with sensor fusion:

Active and dynamic information fusion for multi-sensor systems with dynamic bayesian networks.

Hope it helps.


What is slowing down classic DBSCAN algorithm How to apply CSR Matrix on DBSCAN algorithm in python without using any libraries?

Update: Matrix size (8580, 126356)

I have given a shot and implemented the algorithm. It runs rather slow. I guess its because regionQuery function calculates the euclidean distance  between a chosen point and every other point in the dataset.

Is there any way this problem can be solved?

Write the DBSCAN code yourself.
Run the code.
Observe that your code likely will be a lot slower than the libraries code.

It also depends on the data spread. When I had data between 0-1 it worked well.
When I tried to threshold some data inside so I had many samples with very small values.
It started to work slowly.


Data scaling before or after PCA I have seen senior data scientists doing data scaling either before or after applying PCA.

What is more right to do and why?

I once heard a data scinetist state at a conference talk: "Basically, you can do what you want, as long as you know what you are doing."

This also applies here. The more statistically sound way would be to transform all variables prior to additional steps such as PCA or factor analysis. Then you still know the scale of your variables and can interpret the rescaling in the context of your application. If you have no such interpretation, but good reasons for rescaling your principal components due to computational issues arising if some values are to close to zero while others are quite large, rescaling the components makes sense. However, reversing this process and still being able to interpret the effect of the rescaling operation in your context will become almost impossible.

"It results more important to balance the classes rather than reduce the dimensionality, at least in terms of accuracy; (ii) The best choice seems to be the application of SMOTE followed by PCA.."

Link: https://core.ac.uk/download/pdf/61408511.pdf


Launch Orange3 core dump for any >3.17 Updated to 3.19 using Anaconda navigator, from CLI I receive these messages

orange-canvas
realloc(): invalid old size
Aborted (core dumped)

python3 -m Orange.canvas
Segmentation fault (core dumped)

Using Anaconda navigator got this log

/home/davide445/.anaconda/navigator/scripts/orange3.sh: line 3:  8261 Aborted                 (core dumped) orange-canvas > /home/davide445/.anaconda/navigator/scripts/orange3-out-1.txt 2> /home/davide445/.anaconda/navigator/scripts/orange3-err-1.txt

Same problem with 3.20.1 or 3.18, is working using 3.17. Working on Kubuntu 18.04 LTS.
Any suggestion for troubleshooting, many thanks.

I have the same problem as you, and I just used pip to reinstall the 3.17 version.

pip install Orange3==3.17


Using the GA R package to optimize the weights of a MLP neural network The neural network I am trying to evolve uses the tanh as an activation function in each neuron and has a topology of 1-5-1, so I need at least 5 weights. The solution of the GA is a real-number vector of length 5, which represents the weights of the network and each weight should take values between -5 to 5. I wrote an R function to use as a fitness function which returns the mean squared error (MSE) of the output data in comparison to the desired output. I want it to learn the cubic function. The input data I am using is

input<-seq(-1,1,0.02)

and the output data is its cubic function

des_out<-input^3

The evaluation function is the following

evalnn<-function(x){

mse<-0
for(i in 1:length(input)){

     nn.out <- tanh(x[1]*input[i]) + tanh(x[2]*input[i]) + tanh(x[3]*input[i]) + tanh(x[4]*input[i]) + tanh(x[5]*input[i])
     mse <- (des.out[i] - nn.out)^2 + mse

}
return(-(mse)/length(input))
}

I have set the return value to negative, because I want the smallest value to be thought as the best fit.

gann<-ga(type="real-valued", fitness=evalnn,min=c(-5,-5,-5,-5,-5),max=c(5,5,5,5,5),popSize=100,maxiter=150,pmutation=0.01,pcrossover=0.8)

What I always get back from the GA are weights that make up a linear function although I have been experimenting with the ga's parameters quite a lot, i.e. I have tried all crossover methods and mutations. https://cran.r-project.org/web/packages/GA/GA.pdf.

The cubic function plotted together with my output function.

I have used a linear function as training data before the cubic with this implentation and worked. It has trouble with the non-linear.

If anybody could figure out why do I get this, is there something I have missed.
Thank you

I finally got the results I was looking for.

Apparently a single hidden layer neural network cannot learn the cubic function. Instead of just evolving the weights, I included the topology of the neural network into the chromosome. My neural network can have maximum 5 hidden layers with maximum 5 neurons in each layer and the chromosome contains information for the structure of the network and the selection of the weights.

When I ran the GA with population of 200 and for 100 iterations I got a full topology of:

1(input) - 4-2-3-2-2 - 1(output), a mean squared error of 0.0051 and the following plot

I would be glad if anyone has any comments or questions.


How can both generator and discriminator losses decrease? In this paper there is a plot of how the loss of gans looks through epochs.

Figure 2:

These are of course averaged losses.

How can both the discriminator loss and generator loss decrease?

real_y = discriminator(real_sample)
fake_y = discriminator(generator(noise))
discriminator_loss = real_y-fake_y+1 # nicer, it is between 0 and 2

fake_y = discriminator(generator(noise))
generator_loss = fake_y

I would expect one of the losses to increase as the other one decreases. Since they use the same calculation of fake_y and one decreases -fake_y and the other fake_y. One optimizer is making fake_y less and the other optimizer is making it more.

Maybe the loss functions aren't calculated like I said.

In the widely used analogy:

  In simple terms the generator is like a forger trying to produce some counterfeit material, and the discriminator is like the police trying to detect the forged items.

We can measure how good the police is by how many times out of 100 fakes he can identify the real and fake ones and the forger as deceiving the policeman out of 100 fakes.

Wouldn't that mean that if the police get better, the forger gets worse? (if using the aforementioned measure of how good they are)

Therefore we wouldn't be able to see both of them being good at the same time! But the graph from the paper indicates otherwise.

What am I missing?

In the subsection of the figure, they compute the average loss across 100 iterations, which is why the loss is monotonically decreasing because on average the loss does decrease with the training.
You are correct in inferring that if this was reported on an iteration to iteration basis, the loss would be a zig zag curve, which is less nice to look at than a smooth curve.
As in the comments, the loss can monotonically decrease to 0, if it is the Wasserstein distance. The Wasserstein distance is computed as $\sup_{f \in L_{k}} E_{x \sim P_X}[f] - E_{x \sim P_{\theta}} [f]$ where the first term is the expectation of the function, $f$ (discriminator) over the batch and the second term is the generated data from the generator computed on the discriminator. Note that by this formulation, it returns the $k$-Wasserstein distance.

This is because the discriminator in WGan isn't a classifier and works differently from the traditional GAN.
You are 100% correct that in a normal GAN it wouldn't make any sense for both losses to decrease as shown.


What is an appropriate machine learning technique to analyse development of status over time? I have a dataset as follows (not the actual data, but representative):

User   Age  Salary  Status_2000  Status_2001  Status_2002  Status_2003  ... Status_2019
John   30   10000   A            A            A            B            ...  A
Mary   25   20000   A            B            B            B            ...  C
Jacob  35   30000   E            F            F            F            ...  A
Jenny  28   22000   A            F            F            A            ...  F

I have over 50,000 rows and I want to perform an analyse the development patterns of the Status from 2000 to 2019. However, the Status are encoded with letters from A to J. I would like to ask whether there are any machine learning methods or clustering methods which could help in this analysis? I initially thought of K-Means but I'm not sure if that is appropriate even with Label Encoding.

In the same dataset, there are also the following columns:

User   Age  Salary  2000_%  2001_%  2002_%  2003_%  ... 2019_%
John   30   10000   0%      0%      20%     25%     ... 100%
Mary   25   20000   0%      10%     40%     60%     ... 80%
Jacob  35   30000   0%      80%     100%    80%     ... 80%
Jenny  28   22000   0%      0%      0%      20%     ... 60%

where the data represents % of the maximum amount from 2000 to 2019, and I want to perform an analysis on the development of the %. Are there any techniques suitable for such analysis?

Thank you all for your help in advance!

Yes, K-means is an option. But you need to do some data processing first.

My first concern is about quantifying those statuses (are they linearly depended, like does B means something in the middle of A and C)? then numerate them (A to 1 and B to 2 and C to 3 etc)
Represent your data set in a dimensional way, melt years columns into three columns

Year (with values of 2000, 2001, 2003 etc) so you get a time series dimension.
Year_status (status value).
Year_percentage (percentage of the year).

Try to treat pattern by moving window (kernel), and try different sizes.. for example size of 3 will cluster every 3 followed statuses into patterns centroids (for example.. having a common occurrence of A, E, A)


Multiclass classification with large number of classes but for each user the set of target classes is known Hi Data Science Stack Exchange! I'm new here but I'm familiar with some machine learning theory (took some courses in school) and my question is more about how to apply ML in a practical setting.

I have this project where I'm trying to design a system to predict which "store" a user is going to buy a given item from. However, the set of stores a user can potentially buy from is already known for each user (because this promotion only works at limited stores they signed up for). On average, the set of stores the average user can buy from is around 3 but the # of distinct stores across all users is about 10000.

In theory, for a single user this seems like a simple classification problem. We have historical information such as the time/day/month the user bought item Y from store X along with other features such as location (postal code) where the user lives and features related to the type of item they are buying (cost, weight, brand).

However, the issue is that there's currently around 6000 users so going with this approach seems like I would need a separate model for each user but that doesn't seem like an efficient solution to me or at least not how I've generally seen ML algorithms used. Unfortunately, I don't see any other way I could take into account the fact that the set of stores a user can already buy from is known for each user already. I suppose I could have a categorical variable for each buy-able store as a feature but then that would be equivalent to adding 10000 features and I'm not sure if this would scale as the number of distinct stores increases.

It would be really helpful if anyone has any insight on how to apply machine learning techniques to this kind of problem in general as this is sort my first time working on a "real" problem. Thanks!

so it sounds like you have historical transaction data. If you do, you probably want to train a model using store label as your dependent variable, and all the others independent variables.

set.seed(100)
items <- c("milk", "cheese", "steak", "apple", "eggs")
df <- data.frame(trans_id = 1:100,
                       cust_id = sample(1:10, 100, replace = TRUE),
                       item = sample(items, 100, replace = TRUE)
                       )

df$store <- as.factor(sapply(df$cust_id, function(x) {
    sample(letters[cust_id:(cust_id+2)],1)
  }))

head(df)
##  trans_id cust_id   item store
##1        1       4 cheese     e
##2        2       3 cheese     f
##3        3       6   milk     f
##4        4       1 cheese     f
##5        5       5  steak     d
##6        6       5  apple     f

#train General linear model
glm(formula = store ~., family = binomial(link = "logit"),
     data = df)

##all:  glm(formula = store ~ ., family = binomial(link = "logit"), data = df)
##
##Coefficients:
##(Intercept)     trans_id      cust_id   itemcheese
##   1.320095    -0.007399    -0.066363     0.024632
##   itemeggs     itemmilk    itemsteak
##   0.084018     0.047022     1.069399

library("e1071")
#train naive bayes
naiveBayes(formula = store ~., data = df)

Depending on business constraints you may or may not want to include customer id as a predictor variable (include if you will only be predicting stores for a finite known pool of customers; don't include if your model needs to generalize across customers). Similarly, if this were a real data set, you probably wouldn't include a transaction id as a predictor variable, because it won't generalize to new cases (transaction ids are unique)

Another approach would be to add a feature that captures which stores the customer was eligible to avail discounts at. A classification model given a large enough dataset would probably figure out that connection:)


Publicly Available Datasets One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level analysis. Looking at the other people's effort, especially other questions on this site, it appears that many people in this field are doing somewhat repetitive work. For example analyzing tweets, facebook posts, Wikipedia articles etc. is a part of a lot of big data problems.

Some of these data sets are accessible using public APIs provided by the provider site, but usually, some valuable information or metrics are missing from these APIs and everyone has to do the same analyses again and again. For example, although clustering users may depend on different use cases and selection of features, but having a base clustering of Twitter/Facebook users can be useful in many Big Data applications, which is neither provided by the API nor available publicly in independent data sets.

Is there any index or publicly available data set hosting site containing valuable data sets that can be reused in solving other big data problems? I mean something like GitHub (or a group of sites/public datasets or at least a comprehensive listing) for the data science. If not, what are the reasons for not having such a platform for data science? The commercial value of data, need to frequently update data sets, ...? Can we not have an open-source model for sharing data sets devised for data scientists?

There is, in fact, a very reasonable list of publicly-available datasets, supported by different enterprises/sources.

Some of them are below:

Public Datasets on Amazon WebServices;
Frequent Itemset Mining Implementation Repository;
UCI Machine Learning Repository;
KDnuggets -- a big list of lots of public repositories.

Now, two considerations on your question. First one, regarding policies of database sharing. From personal experience, there are some databases that can't be made publicly available, either for involving privacy restraints (as for some social network information) or for concerning government information (like health system databases).

Another point concerns the usage/application of the dataset. Although some bases can be reprocessed to suit the needs of the application, it would be great to have some nice organization of the datasets by purpose. The taxonomy should involve social graph analysis, itemset mining, classification, and lots of other research areas there may be.

Now google also provides a dataset search!!

https://toolbox.google.com/datasetsearch

It gives you link to all the other sites. It's like one search to search them all!!


How do we define a linearly separable problem? When we talk about Perceptrons, we say that they are limited for approximating functions that are linearly separable, while Neural Networks that use non-linear transformations are not.

I am having trouble understanding this idea of linear separability. Specifically, does it apply only for binary classification or it generalizes for N classes? If so, how does a linear decision boundary for lets say 4 classes even look like?

1.) Perceptron is a non-linear transformation!
2.) Linear seperable function is only defined for boolean functions, see Wikipedia.
Therefore, yes, the statement is meant only for binary classification.
3.) For general functions, see the universal approximation theorem.

I will try to explain this problem in a classification scenario. Say we begin with two classes of images say dog and cat. And we are asked to classify the images whether the given test image is dog or cat. Imagine the images that we are given is $8 \times 8$. So each image is represented by an $8 \times 8=64$ dimensional vector. For the training dataset, one such training image will form a data point in this 64-dimensional space. After lots of training data of cat and dog images say 1000 training cat images and 1000 training dog images, we get 2000 such points in the 64-dimensional space. If in that space, the points can be classified using a 64-dimensional hyperplane, then it's linearly separable in an easy way. If not, we can use kernel which is a nonlinear function of the given image, which will transfer the 64-dimensional space into a higher space (more than 64, where we can find linear separability. In a word, each problem is linearly separable. For more information, you can go through Linear Support Vector Machine and Kernel Support Vector Machines.


Deep learning error: All layer names should be unique I am working in deep learning project using vgg16.
I got the following error and I could not solve it.

RuntimeError: ('The name "block1_conv1" is used 2 times in the model. All layer names should be unique.

The code:

vgg16_model = keras.applications.vgg16.VGG16()

input_layer1 = vgg16_model.input
vgg16_model.get_layer(index = 0).name = 'input1'
last_layer1 = vgg16_model.get_layer('flatten').output

# Adding layers
x = Dense(2048, activation='relu', name='fc1')(last_layer1)
x = BatchNormalization(name='bn1')(x)
fc12 = Dense(2048, activation='relu', name='fc2')(x)
bn12 = BatchNormalization(name='bn2')(fc12)
out1 = Dropout(0.4, name='dropout1')(bn12)

#Classification layer
output_layer1 = Dense(no_classes, activation='softmax', name='prediction1')(out1) #106 classes
model1 = Model(input_layer1, output_layer1)

#Train the last block
model1.trainable = True
set_trainable = False
for layer in model1.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

vgg16_model2 = keras.applications.vgg16.VGG16()

input_layer2 = vgg16_model2.input
vgg16_model2.get_layer(index = 0).name = 'input2'

# Remove layers from block4_pool layer
last_layer2 = vgg16_model2.get_layer('block4_pool').output

# Adding layers
x = Conv2D(512, (3, 3), activation='relu', name='block5_conv1', padding='same')(last_layer2)
x = BatchNormalization(name='bn1')(x)
x = Conv2D(512, (3, 3), activation='relu', name='block5_conv2', padding='same')(x)
x = BatchNormalization(name='bn2')(x)
x = Conv2D(512, (3, 3), activation='relu', name='block5_conv3', padding='same')(x)
x = BatchNormalization(name='bn3')(x)
x = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), name = 'block5_pool')(x)
x = Flatten(name = 'flatten')(x)
x = Dense(2048, activation='relu', name='fc1')(x)
x = BatchNormalization(name='bn4')(x)
fc22 = Dense(2048, activation='relu', name='fc2' )(x)
bn25 = BatchNormalization(name='bn5')(fc22)
out2 = Dropout(0.3, name='dropout1')(bn25)

#Classification layer
output_layer2 = Dense(106, activation='softmax', name='prediction2')(out2)
model2 = Model(input_layer2, output_layer2)

#Train the last block
model2.trainable = True
set_trainable = False
for layer in model2.layers:
    if layer.name == 'block5_conv1':
        set_trainable = True
    if set_trainable:
        layer.trainable = True
    else:
        layer.trainable = False

con = concatenate([out1, out2])  # merge the outputs of the two models
output_layer3 = Dense(no_classes, activation='softmax', name='prediction3')(con)
multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer3])

I tested model1 and model2, and I got good results. But when I tried to merge the two models the error appear. The problem initiated here:

multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer3])

Please help to solve this problem.

The problem is, you are trying to instantiate two VGG16 models at the same time and its confusing for the kernel to figure out which graph it needs to use. Atleast that is the speculation I have, because of the problems faced when trying to load multiple models in keras with tensorflow backend, its not straight forward.

Try commenting the code from when you are instantiating the second model and this error should go away.


Is it ok to trust regression predictions when none of the coefficients are statistically significant? Background to the problem: I am estimating individual treatment effects using double machine learning model. I do not know true treatment effects for my problem.

Double ML: Given Y (outcome), T (treatment) and X ( features)

Y = aT + bX + error

coefficient a is of interest (measures treatment effect) .

Double ML procedure:

Fit Y ~ X  => Compute residuals (Y* = Y – Y’) – Residuals are
treatment effects to be estimated
Fit T ~ X => Compute residuals (T* = T- T’) – This model captures
variation in T explained by X
Fitting a model (Y* ~ T* ) on residuals will give the average
treatment effects

I am fitting a linear regression model (Y* ~ T* ) and none of the coefficients are statistically significant. Instead of relying on point estimates, I am computing prediction confidence intervals and p-value to check if the predicted value is statistically significant or not.

Is this approach good?

It depends on the goal of the project.
Statistical significance is more important for publishing in academic journal. For applied projects, statistical significance is less important.
Even if a model is not statistical significant, it can have value. The model may summarize the data and identify relationships. The model maybe practically important, the effect maybe of value to practitioners.


I have few basic questions on R Stem and Leaf Plot This data when I plot - 60 85 72 59 37 75 93 7 98 63 41 90 5 17 97

stem(data)

 0 | 577
 2 | 7
 4 | 19
 6 | 0325
 8 | 50378

Now If I reconstruct data it is 5, 7, 7, 27, 60, 63, 62, 65, 85, 80, 83, 87, 88

This does not produce the input in reverse.

With the default scale, you see that numbers left of the bar go up by two - hence anything after the 4 is forty-something or fifty-something:

> stem(d,scale=1)

  The decimal point is 1 digit(s) to the right of the |

  0 | 577
  2 | 7
  4 | 19
  6 | 0325
  8 | 50378

Using scale=2, you'll see numbers left of the bar go up by one, so now you can get exact reconstruction of your input, since you input integers:

> stem(d,scale=2)

  The decimal point is 1 digit(s) to the right of the |

  0 | 57
  1 | 7
  2 |
  3 | 7
  4 | 1
  5 | 9
  6 | 03
  7 | 25
  8 | 5
  9 | 0378

Going further you can even split it by first and second five within each decade:

> stem(d,scale=4)

  The decimal point is 1 digit(s) to the right of the |

  0 | 57
  1 |
  1 | 7
  2 |
  2 |
  3 |
  3 | 7
  4 | 1
  4 |
  5 |
  5 | 9
  6 | 03
  6 |
  7 | 2
  7 | 5
  8 |
  8 | 5
  9 | 03
  9 | 78

Stem plots do not always guarantee you can reproduce the data by reversing the process, and that's not what they're for.

try this
S=c(60, 85, 72, 59, 37, 75, 93, 7, 98, 63, 41, 90 ,5 ,17 ,97)
;stem(S,scale = 2)
A value of scale = 2 will cause the plot to be roughly twice as long as the default.


Feature selection for regression Suppose I have a response variable y and and a set of feature variables (x1, x2 ... xn). I wish to find which of x1...xn are the best features for y in a regression problem (the relationship might not be linear).
Is there any way I can do this kind of feature selection without using any correlation measure or regression function in the process (i.e. I cannot use any filter or wrapper methods)?

You can train a LightGBM Regressor. LightGBM Regressor feature selection methods embbed in it. You can direct plot them to see which features are important.
See this link.
LightGBM Plot Importance

Google up scikit learn feature selection. You should look to use the f_regression and mutual_info_regression to identify the best features for the problem at hand.


Recognising made up terms Say I have a tagging system on an electrical circuit:

Name          Description
-------       --------------
BT104         Battery. Power source
SW104         Circuit switch
LBLB-F104     Fluorescent light bulb
LBLB104       Light bulb
...           ...

I have a hundreds of tags created by people who should have followed my naming conventions but sometimes they make mistakes add unnecessary additional characters to tag names (i.e. BTwq104 etc.).

Up until now I used regular expressions, that I built over time whilst observing various inconsistencies that users introduce whilst naming different parts of their circuits, to parse the names and tell me what the different elements are. For example: name 'BT104' would tell me it's a battery on circuit 104.

I would like to investigate or use a machine learning technique to identify what a tag name is (same way I used regular expressions). Any suggestions and approaches are welcome.

So far I tried Named-entity recognition suggested technique "Bag of words". Followed a few tutorials here and here (latter being the most useful in learning). None of them produced wanted results if any. I think that "Bag of words" are mostly used for real word rather than made up words.

You could treat this as a spelling error identification problem. The "Name" column should be a set of unique keys. You can calculate the Levenshtein distance, which finds the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other, between each key. Then set a similarity threshold. Any two keys that are greater than the similarity threshold are merged together.

You can write your own based on Peter Norvig's Spelling Corrector or use Python's FuzzyWuzzy package. The code would be something like this:

from fuzzywuzzy import process

names  = set("BT104 SW104 LBLB-F104 LBLB104".split())
threshold = 85

for name in names:
    leave_current_out = names - {name}
    for match, score in process.extract(name, leave_current_out, limit=1):
        if score >= threshold:
            print(f'Potential misspelling: {name} with {match}')

You could try a recurrent model to identify the part, such that it reads the strings "letter by letter". I would extract the circuit number and parse that by hand, then feed the textual part into the network. In TensorFlow a basic example could look something like this:

import tensorflow as tf

# Words used by people
TAGS = [
    'BT',
    'SW',
    'LBLB-F',
    'LBLB',
    # ...
]
# "Component id"
LABELS = [
    0,  # 0 => Battery
    1,  # 1 => Circuit switch
    2,  # 2 => Fluroescent light bulb
    3,  # 3 => Light bulb
    # ...
]

# Number of different components
NUM_CLASSES = 4

NUM_LAYERS = 1
LAYER_SIZE = 64
BATCH_SIZE = 100
NUM_EPOCHS = 100

# Inputs must be strings of the same size padded with '\0'
input_tags = tf.placeholder(tf.string, [None], name='Input')
# Convert to ascii values
tag_ascii = tf.decode_raw(input_tags, tf.uint8)
# Get actual lengths
mask = ~tf.equal(tag_ascii, 0)
tag_length = tf.reduce_sum(tf.cast(mask, tf.int32), axis=1)
# Convert to one-hot encoding
tag_1h = tf.one_hot(tag_ascii, 256, dtype=tf.float32)
# RNN
cells = [tf.nn.rnn_cell.BasicLSTMCell(LAYER_SIZE) for _ in range(NUM_LAYERS)]
rnn = tf.nn.rnn_cell.MultiRNNCell(cells)
rnn_output, _ = tf.nn.dynamic_rnn(rnn, tag_1h, sequence_length=tag_length, dtype=tf.float32)
# Get last RNN output
last_rnn_indices = tf.stack([tf.range(tf.shape(rnn_output)[0]), tag_length - 1], axis=-1)
rnn_last_output = tf.gather_nd(rnn_output, last_rnn_indices)
# Output layer
output_weights = tf.get_variable('OutputWeights', (LAYER_SIZE, NUM_CLASSES))
output_logit = rnn_last_output @ output_weights
# Final output as distribution and highest-scoring class
output_dist = tf.nn.softmax(output_logit)
output_class = tf.argmax(output_logit, axis=-1)
# Loss and training
input_labels = tf.placeholder(tf.int32, [None], name='Class')
loss = tf.losses.sparse_softmax_cross_entropy(labels=input_labels, logits=output_logit)
# Choose optimizer and hyperparameters
train_op = tf.train.AdamOptimizer().minimize(loss)
# Variable initialization
init_op = tf.global_variables_initializer()

# Preprocess words so all have the same size
max_tag_len = max(len(tag) for tag in TAGS)
tags_padded = [tag + '\0' * (max_tag_len - len(tag)) for tag in TAGS]

num_examples = len(tags_padded)
with tf.Session() as session:
    session.run(init_op)
    # Train
    for i_epoch in range(NUM_EPOCHS):
        for idx_batch in range(0, num_examples, BATCH_SIZE):
            tags_batch = tags_padded[idx_batch:idx_batch + BATCH_SIZE]
            labels_batch = LABELS[idx_batch:idx_batch + BATCH_SIZE]
            session.run(train_op, feed_dict={input_tags: tags_batch, input_labels: labels_batch})
    # Check results
    predictions, dist = session.run([output_class, rnn_output], feed_dict={input_tags: tags_padded})
    for tag, label, prediction in zip(TAGS, LABELS, predictions):
        print('Tag {} is class {} and was predicted to be class {}.'.format(tag, label, prediction))
    # Test for an unknown tag: 22LBLB-T should be class 3
    tag = '22LBLB-T'
    prediction = session.run(output_class, feed_dict={input_tags: [tag]})[0]
    print('Tag {} was predicted to be class {}.'.format(tag, prediction))

Output:

Tag BT is class 0 and was predicted to be class 0.
Tag SW is class 1 and was predicted to be class 1.
Tag LBLB-F is class 2 and was predicted to be class 2.
Tag LBLB is class 3 and was predicted to be class 3.
Tag 22LBLB-T was predicted to be class 2.

Which basically takes each string, converts it into vectors of numbers, converts these to one-hot encoding and feeds them to a recurrent network (plus an output layer). In this particular case it pads the strings with null characters at the end so all have the same length.

I added a test at the end for an unseen tag 22LBLB-T that should be classified as light bulb. In this case the model failed and said it's a fluorescent light bulb, although to be fair it didn't have many clues to figure out the right answer, given the data (in fact, that tag is in this case more similar to the fluorescent light bulb, since it has a hyphen - you could consider filtering filtering characters like hyphens and others if you think they are going to "confuse" the model). In any case, the prediction by the model "makes sense" with respect to the provided data (it didn't predict it to be a battery or circuit switch, which would not make any sense).


Forecasting/predicting techniques for qualitative data? I have a food alert dataset composed of nominal qualitative variables, such as type of alert, country of origin, action taken, etc. as well as the date on which the alert was recorded.

What techniques are there to predict what type of alert is most likely in a time window when there are only qualitative variables in our dataset?
Or given a variable (for example, country of origin = France) predict the rest of the variables?

I have used machine learning algorithms before but not with datasets composed of nominal qualitative variables.

On the other hand, what are the most appropriate techniques for clustering and classification this kind of datasets?

I know it's a broad question, so I expect broad answers to guide me in a general way

I have about 50000 registered alerts. In the example of the image not all the variables are shown

You can encode the various columns of qualitative data using LabelEncoder(). For example:

label_encoder = LabelEncoder()
risk = label_encoder.fit_transform(df['RISK_DECISION'])
action = label_encoder.fit_transform(df['ACTION_TAKEN])
etc...

Maybe you could start by actually and precisely defining what you want to do with your data.

Apparently you'd like to predict some probability for each type of alert (which I assume is given by the TYPE column), and find which one gets the highest probability (that boiling down to finding what type of alert is the most likely to happen in a given time interval). Well if it's what you want to do, maybe have a look a time series forecasting ? May be an starting point...

Considering the fact that you only have nominal qualitative variables, some kind of one-hot-encoding maybe could help with clustering, for example you could encode each variable for each observation, then compute cosine similarities. That would be a way of getting a glance at a kind of distance between your observations... even it's a very basic idea.


Unable to understand the usage of labels argument in sklearn.metrics.f1_score I am trying to model a dataset with RandomForest Classifier.
My dataset has 3 classes viz. A, B, C. 'A' is the negative class and 'B' and 'C' are positive classes.

In GridSearch I wanted to optimize on F1-score since the number of samples in all the classes are not evenly distributed and class 'A' has the highest number of samples.

That is where I wanted to understand the usage of labels argument.
The doc says:

  labels : list, optional The set of labels to include when average !=
  'binary', and their order if average is None. Labels present in the
  data can be excluded, for example to calculate a multiclass average
  ignoring a majority negative class, while labels not present in the
  data will result in 0 components in a macro average.

I could not understand it properly. Does it mean, In my screnario I should have labels as labels = ['B', 'C'], just the positive class?
Kindly Help

custom_scoring = make_scorer(f1_score, labels=[???],average='weighted')
clf = RandomForestClassifier(class_weight='balanced', random_state=args.random_state)
grid_search = GridSearchCV(clf, param_grid=param_grid, n_jobs=20, scoring=custom_scoring)

The F1 measure is a type of class-balanced accuracy measure - when there are only two classes, it's very straightforward, as there's only one possible way to compute it. With 3 classes, however, you could compute the F1 measure for classes A and B, or B and C, or C and A, or between all three of A, B and C.

It seems that the "labels" parameter is telling the method which classes to compute your measure over. Since F1 is already class-balanced, you probably want to include all three labels for your measure. This parameter is likely more important for imbalance-insensitive measures like raw accuracy, as it will allow you to compute accuracy in a subset of the data - in the documentation example, they use it to exclude a majority class, allowing the user to evaluate the accuracy in the minority only. If you have a huge imbalance, say 99% of your data is of one class, your accuracy measure will be completely dominated by accuracy within that class - for this reason, it might be more informative to see how well the classifier does in the 1% only.

In case of imbalanced dataset, accuracy score of sampling algorithm yields an accuracy of 99% which seems impressive, but minority class could be totally ignored in case of imbalanced datasets.

If data set is imbalanced, pre-processed data set with sampling algorithm (for e.g SMOTE) and re-sample it. It will create equal sets of examples for class based on neighbors.

https://stackoverflow.com/questions/57205718/how-can-we-be-sure-of-the-efficiency-of-a-neural-network/57211888#57211888


Is there a way to; easily, merge data into a CSV (or Excel) file without having to edit each file individually? I have several marketing files that I need to download on a weekly basis.
Let’s say 15 different files. Each file contains the same field names: field 1, field 2, field 3, field 4, field 5, field 6, field 7, field 8, field 9. The data contained in the fields are different and specific to each file.
Each file can contain 10 to 1000s records.
What I need to do, is add 2 additional fields to each record in each file. These fields; field A and field B will contain the same info for every record in each specific file. Each file will contain different info to add.
File 1      Need to add Field A and Field B to every record.
File 2      Need to add Field C and Field D to every record.
File 3      Need to add Field E and Field F to every record.
I can edit each file in Excel, add the columns, name the columns, add the 2 fields to the first row and
then copy and paste to all the other rows.
Is there a way to specify the file, field 1 and field 2 in order to add the info more efficiently and quickly?
Thank you!

You can either load the excel files directly into Python, e.g. using the pandas package, or you can use a package that hits the data where it lives i.e. a package that directly modifies the Excel files.

Python and Pandas

import pandas as pd

data = pd.read_Excel('/path/to/your/file)

Data is basically a table that will look like the Excel itself. You can add fields (usually referred to as columns) like so:

data['new field'] = new_data

new_data will contain all values for all rows in the table, e.g. as a list:

new_data = [12, 24, 13, 66, 72, 45]    # assuming there are 6 rows (a.k.a. samples)

You will have to alter the code for each of your files. You will also need to think about how to get your new data into that list (or a similar construction).

Caveat: this way might have problems if you have some very specific formatting or special Excel features in use (more than just conditional formatting - think dropdown lists and hidden/folded rows and columns).

Excel interface

There are a few good packages that will allow you to speak directly with the excel file and make use of things such as sheets/tabs and references to cells e.g. A1 is the top left cell in Excel.

Have a look at the XlsWriter package or the python-excel package. Both let you access the fill and add/remove data as you please. I have used both for simple tasks and they both got the job done quite nicely.

Follow those links to see some nice examples of how to manipulate Excel programmatically.

It is worth mentioning that the first option above, using something like Pandas, will offer a lot more functionality and would likely not present a steeper learning curve (in my opinion!).

merging csv files is possible:

make sure that you have all the csv files in the same folder.
open that folder in a cmd window.
use the following command:
copy *.csv combined.csv

this command will copy all the csv file into a single file called combined.csv

If you are looking for more advanced solutions, you can search for free packages and tools to do these things as proposed in previous answer.


Interpreting Decision Tree in context of feature importances I'm trying to understand how to fully understand the decision process of a decision tree classification model built with sklearn. The 2 main aspect I'm looking at are a graphviz representation of the tree and the list of feature importances. What I don't understand is how the feature importance is determined in the context of the tree. For example, here is my list of feature importances:

Feature ranking:
1. FeatureA (0.300237)

FeatureB (0.166800)
FeatureC (0.092472)
FeatureD (0.075009)
FeatureE (0.068310)
FeatureF (0.067118)
FeatureG (0.066510)
FeatureH (0.043502)
FeatureI (0.040281)
FeatureJ (0.039006)
FeatureK (0.032618)
FeatureL (0.008136)
FeatureM (0.000000)

However, when I look at the top of the tree, it looks like this:

In fact, some of the features that are ranked "most important" don't appear until much further down the tree, and the top of the tree is FeatureJ which is one of the lowest ranked features. My naive assumption would be that the most important features would be ranked near the top of the tree to have the greatest impact. If that's incorrect, then what is it that makes a feature "important"?

It is not necessary that the more important a feature is then the higher its node is at the decision tree.

This is simply because different criteria (e.g. Gini Impurity, Entropy-Information Gain, MSE etc) may be used at each of two these cases (splitting vs importance).

For example, at SkLearn you may choose to do the splitting of the nodes at the decision tree  according to the Entropy-Information Gain criterion (see criterion & 'entropy' at SkLearn) while the importance of the features is given by Gini Importance which is the mean decrease of the Gini Impurity for a given variable across all the trees of the random forest (see feature_importances_ at SkLearn and here).

If I am right, at SkLearn the same applies even if you choose to do the splitting of the nodes at the decision tree according to the Gini Impurity criterion while the importance of the features is given by Gini Importance because Gini Impurity and Gini Importance are not identical (see also this and this on Stackoverflow about Gini Importance).

Variable importance is measured by decrease in model accuracy when the variable is removed. The new decision tree created with the new model without the variable could look very different to the original tree. Splitting decision in your diagram is done while considering all variables in the model.

What variable to split at the root (and other nodes) is measured by impurity. Good purity (e.g: everything in the left branch has the same target value) is not a guarantee for good accuracy. You data might be skewed, your right branch have more responses than your left branch. Therefore, it's no good just correctly classify the left branch, we also need to consider the right branch as well. Therefore, the splitting variable might or might not be an important variable for overall model accuracy.

Variable importance is a better measure for variable selection.


Develop a marketing look-alike profile for prospective customers I work for a business-to-business company that has a large database of existing clients (small businesses) with various columns of data (e.g., industry, credit-worthiness, financial data, etc) describing each client. There are several marketing data vendors I know of that maintain small-business databases with columns of data similar to what my company has.

Question: Using the data my company already has, how I can I identify which customers in the vendors' databases have the highest potential of becoming a customer if marketed to? Should I build a predictive model? Should I do some sort of clustering?

The problem is that you only have positive instances (businesses who have become a customer) but no negative instances (businesses who haven't become a customer).  This prevents use of supervised learning.  You could try unsupervised learning -- e.g., a one-class classifier -- but I don't expect it to be particularly effective.

If you had data on businesses who you tried approaching, but ultimately did not become a customer of yours, then you could try applying supervised learning.

I think that your problem would be well suited for Decision Tree Classification.  The presentation that I linked to should give you a good overview to learn about decision tree models.

This Python tutorial will teach you how to build your own decision tree model from scratch.  With this knowledge, and through searching many more decision tree examples online, I'm sure you'll find yourself in a good spot!


Regex remove 1-2 character sequences: hyphens literal, not word boundaries I need a regex in R to exclude 1 or 2-character words, but which does not treat hyphens as word boundaries.  Here is an example:

z <- c("regulatory protein SR-B1 na na na na", "Dr Foo is na Editor-in-chief", "na P3 protein is popular na na")

The intended 'gsub' would produce the result

  [1] "regulatory protein SR-B1" " Foo Editor-in-chief"     "P3 protein  popular"

This is not produced from the gsub statements below. Instead:

A - 2-character segments unintentionally dropped from hyphenated words (and hyphens kept).

B - 2-character segments unintentionally dropped within hyphenated words (hyphens also dropped):

C - Produces: Error: '-' is an unrecognized escape in character string starting ""
*\b[[-"

D - Nothing dropped or changed:

gsub(" *\\b[[:alpha:]]{1,2}\\b *", " ", z) # A

gsub(" *\\b[-[:alpha:]]{1,2}\\b *", " ", z)  # B

gsub(" *\\b[[\-][:alpha:]]{1,2}\\b *", " ", z)  # C

gsub(" *\\b[[\\-][:alpha:]]{1,2}\\b *", " ", z) # D

Input would be much appreciated.


Cross validation Vs. Train Validate Test I have a doubt regarding the cross validation approach and train-validation-test approach.

I was told that I can split a dataset into 3 parts:

Train: we train the model.
Validation: we validate and adjust model parameters.
Test: never seen before data. We get an unbiased final estimate.

So far, we have split into three subsets.  Until here everything is okay. Attached is a picture:

Then I came across the K-fold cross validation approach and what I don’t understand is how I can relate the Test subset from the above approach. Meaning, in 5-fold cross validation we split the data into 5 and in each iteration the non-validation subset is used as the train subset and the validation is used as test set. But, in terms of the above mentioned example, where is the validation part in k-fold cross validation? We either have validation or test subset.

When I refer myself to train/validation/test, that “test” is the scoring:

Model development is generally a two-stage process. The first stage is training and validation, during which you apply algorithms to data for which you know the outcomes to uncover patterns between its features and the target variable. The second stage is scoring, in which you apply the trained model to a new dataset. Then, it returns outcomes in the form of probability scores for classification problems and estimated averages for regression problems. Finally, you deploy the trained model into a production application or use the insights it uncovers to improve business processes.

As an example, I found the Sci-Kit learn cross validation version as you can see in the following picture:

When doing the splitting, you can see that the algorithm that they give you, only takes care of the training part of the original dataset. So, in the end, we are not able to perform the Final evaluation process as you can see in the attached picture.

Thank you!

scikitpage

If k-fold cross-validation is used to optimize the model parameters, the training set is split into k parts. Training happens k times, each time leaving out a different part of the training set. Typically, the error of these k-models is averaged. This is done for each of the model parameters to be tested, and the model with the lowest error is chosen. The test set has not been used so far.

Only at the very end the test set is used to test the performance of the (optimized) model.

# example: k-fold cross validation for hyperparameter optimization (k=3)

original data split into training and test set:

|---------------- train ---------------------|         |--- test ---|

cross-validation: test set is not used, error is calculated from
validation set (k-times) and averaged:

|---- train ------------------|- validation -|         |--- test ---|
|---- train ---|- validation -|---- train ---|         |--- test ---|
|- validation -|----------- train -----------|         |--- test ---|

final measure of model performance: model is trained on all training data
and the error is calculated from test set:

|---------------- train ---------------------|--- test ---|

In some cases, k-fold cross-validation is used on the entire data set if no parameter optimization is needed (this is rare, but it happens). In this case there would not be a validation set and the k parts are used as a test set one by one. The error of each of these k tests is typically averaged.

# example: k-fold cross validation

|----- test -----|------------ train --------------|
|----- train ----|----- test -----|----- train ----|
|------------ train --------------|----- test -----|

Excellent question!

I find this train/test/validation confusing (I've been doing ML for 5 years).
Who says your image is correct? Let's go to an ML authority (Sk-Learn)
In general, we do k-Fold on train/test (see Sk-Learn image below).

Technically, you could go one step further and do a Cross Validation on everything (train/test/validation). I've never done it though ...
Good luck!


Why 100% accuracy on test data is not good? I was asked this question in an interview and wasn’t able to give a satisfactory answer not only upto the interviewers' expectations but of my own as well. The question was as above only, he later gave an example as if why if my model predicted the prices of oil of tomorrow 100% accurately why that might be bad or why having a model 100% accurate bad or is it? Is there something in the question or is there a deeper explanation?

I see two ways to go:
1 - There is an error
2 - There is no error.

1 - Look for the error

You probably have commited Data Leakage.  You have added the target in one of the features and the model found out.
The validation is not right, you have a time series and you have done random validation.
Your test has only a few instances or it is unique.
The test is repeated from the train.

2 - There is no error

If the prediction is right and you have 100% accuracy, then no need to do Machine Learning. Open the model find where is taking the decision and don't do machine learning, do classical modeling.

For example if your model is a decision tree, just plot it or print it and get the decision and put them yourself.

This sometimes happens when modeling a previous developed algorithm. The new ML model is able to learn what was going on before.

Training a heavy model on a small dataset, all from the same distribution, could cause an overfit. Even on the test set. When you deploy that model on real world data, you may experience a model drastic performance drop. (ie: getting high variance).

Edited: Rethinking the question again and this is what I was trying to explain. Imagine you're trying to train an bird classifier that can predict if there's a bird in a picture or video frame. You train and test your model with high quality and clear images of bird maybe surfed from the internet of taken professionally,.. You get an 100% accuracy on train and test set... But when you deploy your model... It may not perform as well because the images the model would be using may not be as clear as the training and test data.
100% test accuracy isn't bad but not a final performance metric...


How should one deal with null values in time stamp data types? I have a data set where certain features are time stamps for certain important events like time stamp of when a user logged in etc. How should I deal with null values in such cases. Should one just replace null values with current or most recent time stamp. Are there any standard techniques followed ?

Leave those time stamps null in order to indicate that you do not possess the time of those events. Everything else depends on the logic of your application.

It is not clear from your question what your use case is exactly. But for cases like what you mentioned, it is usually more helpful if you generate time series out of each feature. For example, if you have user's login and logout time, you can generate a time series where it is 1 when the user is logged in and 0 when the user is logged out.
But again, tell us more about your use case first.


How to visualize results/errors of multilabel classifiers? For multiclass classification you would normally choose a confusion matrix to plot the error of predicted classes against the target classes.

What is the best way to visualize errors of multilabel classifiers? As multiple classes are predicted at once a mapping of prediction against target is not always possible so confusion matrices are generelly not suitable.

My first idea is to plot a bar chart where each class has a bar for missed and one for false predictions. But is there any standard for visualizing the errors which includes more information just like a confusion matrix for multiclass problems?

There is something called macro-averaging and micro-averaging.

Macro-averaging is just the TPs, TNs, FPs, and FNs of each label class summed up and averaged. You should use macro-averaging for generalizing and evaluating on the overall. You should not use it to make specific decisions.

See details of micro-averaging and more evaluation techniques in the link below.
https://towardsdatascience.com/journey-to-the-center-of-multi-label-classification-384c40229bff#6aaa

I found classification_report to be extremely helpful in understanding how my model is doing for each label. It generates a report detailing the f1-score, recall and precision for each label.


Is There a Way to Re-Calibrate Predicted Probabilities After Using Class Weights? I have classification data with far more negative instances than positive instances. I have used class weights in my models and have achieved the discrimination I want but the predicted probabilities from the models do not match the actual probabilities in the modeling data.

Is there a way to adjust the predicted probabilities from the class weighted models to match the actual probabilities in the data? I have seen equations for under-sampling (https://www3.nd.edu/~rjohns15/content/papers/ssci2015_calibrating.pdf) but they don't seem to work for class weights. I have searched online for an answer but maybe I am not using the right language?

Thank you!

A more general adjustment for resampling (not just the simple undersampling in your linked paper) exists:

Add $\ln\left(\frac{p_1(1-r_1)}{(1-p_1)r_1}\right)$ to the log-odds of each prediction, where $p_1$ is the proportion of the positive class in the original dataset, and $r_1$ is the proportion of the positive class in the resampled dataset.

Equivalently, multiply the odds by the quantity inside the logarithm.  (Unfortunately, this doesn't lead to a clean adjustment directly to the probabilities.)

Let's do a little rewriting to see the connection to your linked paper.  $1-r_1$ is the proportion of negative classes call it $r_0$, and similarly with $p_1$.  Use capitals $R_1, \dotsc$ to denote the number (or total weight) of samples rather than the proportions, and without subscripts $P,R$ to denote total numbers (or weights) of samples before and after resampling.  So the multiplier becomes
$$\frac{p_1(1-r_1)}{(1-p_1)r_1} = \frac{p_1 r_0}{p_0 r_1} = \frac{(P_1/P) (R_0/R)}{(P_0/P) (R_1/R)} = \frac{P_1 R_0}{P_0 R_1}.$$
In the context of the linked paper, positive class samples are not resampled, so $P_1=R_1$ and the adjustment simplifies to $R_0/P_0$, which is the parameter $\beta$ used in the paper.

Finally, using their equation (4), we check the change in odds:
$$\text{new odds} = \frac{p}{1-p} = \frac{1}{\frac1p - 1} =
\frac{1}{\frac{\beta p_s−p_s+ 1}{\beta p_s} - 1} =
\frac{\beta p_s}{1-p_s} = \beta\cdot\text{old odds}. $$

So, what about weightings instead of resampling?  Well, class_weights might have different effects in different algorithms, but generally the idea is that (positive) integer values of class_weights should correspond to duplicating the samples that many times, and fractional values interpolate that.  So, it should be about the same to use the multiplicative factor above.  Using the size version rather than the proportion version, we should interpret $R_0$ and $R_1$ as the total weights of the relevant classes.

I've not been able to find a reference for this version, so I put together a short experiment; it seems to verify that this shift works.
GitHub/Colab notebook

Finally, this shift in log-odds will fail to produce properly calibrated probabilities if the classifier is poorly calibrated on the weighted data.  You could look into calibration techniques, from Platt to Beta to Isotonic.  In this case, the shift above is probably superfluous.

The difference between predicted probabilities and actual probabilities is called training error.

There are many ways to reduce training error. Engineering better features and choosing a different machine learning algorithm are the most common.


Is logistic regression actually a regression algorithm? The usual definition of regression (as far as I am aware) is predicting a continuous output variable from a given set of input variables.

Logistic regression is a binary classification algorithm, so it produces a categorical output.

Is it really a regression algorithm? If so, why?

Logistic regression is regression, first and foremost. It becomes a classifier by adding a decision rule. I will give an example that goes backwards. That is, instead of taking data and fitting a model, I'm going to start with the model in order to show how this is truly a regression problem.

In logistic regression, we are modeling the log odds, or logit, that an event occurs, which is a continuous quantity. If the probability that event $A$ occurs is $P(A)$, the odds are:

$$\frac{P(A)}{1 - P(A)}$$

The log odds, then, are:

$$\log \left( \frac{P(A)}{1 - P(A)}\right)$$

As in linear regression, we model this with a linear combination of coefficients and predictors:

$$\operatorname{logit} = b_0 + b_1x_1 + b_2x_2 + \cdots$$

Imagine we are given a model of whether a person has gray hair. Our model uses age as the only predictor. Here, our event A = a person has gray hair:

log odds of gray hair = -10 + 0.25 * age

...Regression! Here is some Python code and a plot:

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

x = np.linspace(0, 100, 100)

def log_odds(x):
    return -10 + .25 * x

plt.plot(x, log_odds(x))
plt.xlabel("age")
plt.ylabel("log odds of gray hair")

Now, let's make it a classifier. First, we need to transform the log odds to get out our probability $P(A)$. We can use the sigmoid function:

$$P(A) = \frac1{1 + \exp(-\text{log odds}))}$$

Here's the code:

plt.plot(x, 1 / (1 + np.exp(-log_odds(x))))
plt.xlabel("age")
plt.ylabel("probability of gray hair")

The last thing we need to make this a classifier is to add a decision rule. One very common rule is to classify a success whenever $P(A) > 0.5$. We will adopt that rule, which implies that our classifier will predict gray hair whenever a person is older than 40 and will predict non-gray hair whenever a person is under 40.

Logistic regression works great as a classifier in more realistic examples too, but before it can be a classifier, it must be a regression technique!

To put it simply any hypothetical function $f$ makes for regression algorithm if $f:X\rightarrow \mathbb{R}$. Thus logistic function which is $P(Y=1|\lambda, x)=\dfrac{1}{1+e^{-\lambda^Tx}} \in [0,1]$ makes for a regression algorithm. Here $\lambda$ is coefficient or hyperplane found from trained datasets & $x$ is a data point. Here, $sign(P(Y=1|\lambda, x))$ is taken as class.


One Hot encoding for large number of values How do we use one hot encoding if the number of values which a categorical variable can take is large ?

In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary features) in the training dataset which will immensely increase the complexity and hence the training time.

So how do we deal with such cases ?

If you really care about the number of dimensions, you still can try to apply a dimensionality reduction algorithm, such as PCA (Principal Component Analysis) or LDA (Linear Discriminant Analysis), after your one hot encoding.

But know that "56 features" isn't really large and it's highly common in the industry to have thousands, millions or even billions of features.

When there are large number of categorical variables, it is advisable to do one versus rest.


How to make a classification problem into a regression problem? I have data describing genes which each get 1 of 4 labels, I use this to train models to predict/label other unlabelled genes. I have a huge class imbalance with 10k genes in 1 label and 50-100 genes in the other 3 labels. Due to this imbalance I'm trying to change my labels into numeric values for a model to predict a score rather than a label and reduce bias.

Currently from my 4 labels (of most likely, likely, possible, and least likely to affect a disease) I convert the 4 labels into scores between 0-1: most likely: 0.9, likely: 0.7, possible: 0.4, and least likely: 0.1 (decided based on how similar the previous label definitions were in their data). I've been using scatter plots with a linear model to try to understand which model would best fit my data and reducing overfitting, but not sure if there's more I can infer from this except that the data has homoskedasticity (I think? I have a biology background so learning as I go):

I'm not sure if there is a more official way I should be approaching this or if this regression conversion is problematic in ways I don't realise? Should I be trying to develop more scores in my training data or is there something else I should be doing?

Edit for more information: The current 4 labels I have I create based on drug knowledge of the genes and the drug data I currently have for each gene, I could incorporate other biological knowledge I have to make further labels I think.  For example, currently the 'most likely' labelled genes are labelled as such because they are drug targets for my disease, 'likely' label because they are genes which interact with drugs to cause a side effect which leads to the disease, and the other 2 labels go down in relevance until there are least likely genes with no drug related or statistical evidence to cause the disease.

So, the direct answer here is clearly NO.

The answer comes from the definitions of classification and regression. In a classification task what a model predicts is the probability of an instance to belong to a class (e.g. 'image with clouds' vs 'image without clouds' ), in regression you are trying to predict continuous values (e.g. the level of 'cloudiness' of an image).

Sometimes you can turn a regression problem into a classification one. For example if I have a dataset of images with labels of their cloudiness level from 0 to 5 I could define a threshold, i.e. 2.5 and use it to turn the continuous values into discrete one, and use those discrete values as classes (cloudiness level < 2.5 equal image without clouds) but the opposite is definitely not possible.

Here's also a link to a similar question Could I turn a classification problem into regression problem by encoding the classes?

To solve the problem of imbalanced classes there are many ways, not just oversampling, you can generate artificial data, add class weights in the loss function, use active learning to gather new data, use models that returns uncertainty score for each prediction (like bayesian networks), I'm sure here is plenty of answers and strategy you can try.

Yes, you can go this route, using regression rather than classification, but you should one-hot encode your classes. This means that your model will have 4 outputs (alternatively, you can think of it as having 4 models). The first output will be the certainty that label1 applies, the second output label2, etc.

for example, if you have 10 datapoints with labels: 1,2,3,4,2,4,3,1,1,2, your "one-hot" encoded labels look like this:

1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
0 1 0 0
0 0 0 1
0 0 1 0
1 0 0 0
1 0 0 0
0 1 0 0

And a prediction for one data point could look like this:

0.445 0.129 1.234 -0.231

This datapoint has a high probability of having label3, but also a little probability of having label1.


Pandas: how can I create multi-level columns I have a pandas DataFrame which has the following columns:

n_0
n_1
p_0
p_1
e_0
e_1

I want to transform it to have columns and sub-columns:

0
    n
    p
    e
1
    n
    p
    e

I've searched in the documentation, and I'm completely lost on how to implement this. Does anyone have any suggestions?

Finally, I found a solution.

You can find the example script below.

#!/usr/bin/env python3
import pickle
import pandas as pd
import itertools
import numpy as np

data = pd.DataFrame(np.random.randn(10, 5), columns=('0_n', '1_n', '0_p', '1_p', 'x'))

indices = set()
groups = set()
others = set()
for c in data.columns:
    if '_' in c:
        (i, g) = c.split('_')
        c2 = pd.MultiIndex.from_tuples((i, g),)
        indices.add(int(i))
        groups.add(g)
    else:
        others.add(c)
columns = list(itertools.product(groups, indices))
columns = pd.MultiIndex.from_tuples(columns)
ret = pd.DataFrame(columns=columns)
for c in columns:
    ret[c] = data['%d_%s' % (int(c[1]), c[0])]
for c in others:
    ret[c] = data['%s' % c]
ret.rename(columns={'total': 'total_indices'}, inplace=True)

print("Before:")
print(data)
print("")
print("After:")
print(ret)

Sorry for this...

There is a simpler solution:

  data.columns = data.columns.str.split('_', expand=True)

To arrange column names one can also do:

 data.sort_index(axis=1, inplace=True)

To change column levels:

 data = data.reorder_levels([1,0], axis=1)


What is the difference between data singular and data plural ? I don't understand the difference between these terms and google didn't help much. Please answer the question even if you have any slightest idea.

Data is plural, as in 'These data are depicted below'. Singular is 'datum' but it's rare one would have occasion to use the term.

Data (singular): The value of the variable associated with one element of a population or
sample. This value may be a number, a word, or a symbol.
• Data (plural): The set of values collected for the variable from each of the elements belonging
to the sample


Bounding box coordinates prediction How to pass multiple bounding boxes coordinates to CNN model?My goal is to predict the coordinates of texts in an image.

Please look into the Mask R-CNN (Faster CNN)
https://github.com/matterport/Mask_RCNN/


How do I read a dat file, for which I don't know its structure? Is there any way to atleast read the text from the dat file. I have its corresponding mdf file hence I know what all data and columns are there in it.
How do I figure out the contents in my dat file. Cuz all that I am getting currently is some gibberish even if I am opening in binary mode.

from asammdf import MDF
dat_file = r"C:\Users\HPO2KOR\Desktop\Work\data1.dat"
mdf_file = r"C:\Users\HPO2KOR\Desktop\Work\data1.mdf"
df = mdf.to_dataframe()
mdf = MDF(mdf_file)
df.head()

which gives me

How do I read the same data from the dat file, is there any library or code for the same?

And when I am opening the same in binary mode I am getting

data = open(dat_file, "rb")
data.readlines()

which is giving me

b'|CF,2,1,1;|CK,1,3,1,1;\r\n',
 b'|NO,1,7,1,0,,0,;\r\n',
 b'|NL,1,10,1252,0x407;\r\n',
 b'|CT,1,41,0,6,Bench#,24,Korrosionstest 15A046-01,0,;\r\n',
 b'|CT,1,30,0,11,StartOfTest,8,06/30/17,0,;\r\n',
 b'|CT,1,58,0,10,ResultPath,36,c:\\korrosionstest\\daten\\#170161-OR02,0,;\r\n',
 b'|CT,1,59,0,11,GraphicPath,36,c:\\korrosionstest\\daten\\#170161-OR02,0,;\r\n',
 b'|CT,1,31,0,15,GraphicBaseName,5,736_2,0,;\r\n',
 b'|CT,1,26,0,10,PartNumber,5,736_2,0,;\r\n',
 b'|CT,1,31,0,9,VA-Nr. GS,11,170161-OR02,0,;\r\n',
 b'|CT,1,62,0,9,VA-Nr. CC,42,TO_ENV_2017_G2_C1_Platform_CC-122164-03-08,0,;\r\n',
 b'|CT,1,24,0,6,Tester,8,Behrendt,0,;\r\n',
 b'|CT,1,32,0,15,Test Department,6,GS/ETR,0,;\r\n',
 b'|CG,1,5,1,1,1;\r\n',
 b'|CD,1,16,1E-2,1,1,s,0,0,0;\r\n',
 b'|NT,1,27,30, 6,2017,14,25,15.8050001;\r\n',
 b'|CC,1,3,1,1;\r\n',
 b'|CP,1,16,1,2,4,16,0,0,1,0;\r\n',
 b'|Cb,1,33,1,0,1,1,0,11718,0,11718,1,5E-3,0,;\r\n',
 b'|CR,1,30,1,6.103888176768602E-3,0,1,1,A;\r\n',
 b'|CN,1,28,0,0,0,16,ai_iB1_Strom_ECU,0,;\r\n',
 b'|CG,1,5,1,1,1;\r\n',
 b'|CD,1,16,1E-2,1,1,s,0,0,0;\r\n',
 b'|NT,1,27,30, 6,2017,14,25,15.8050001;\r\n',
 b'|CC,1,3,1,1;\r\n',
 b'|CP,1,16,2,2,4,16,0,0,1,0;\r\n',
 b'|Cb,1,37,1,0,2,1,11718,11718,0,11718,1,5E-3,0,;\r\n',
 b'|CR,1,30,1,3.662332906061161E-3,0,1,1,V;\r\n',
 b'|CN,1,31,0,0,0,19,ai_iB1_Spannung_UBB,0,;\r\n',
 b'|CG,1,5,1,1,1;\r\n',
 b'|CD,1,16,1E-2,1,1,s,0,0,0;\r\n',
 b'|NT,1,27,30, 6,2017,14,25,15.8050001;\r\n',
 b'|CC,1,3,1,1;\r\n',

You can try:
np.fromfile(yourfilepath, dtype="byte")

to read content from .dat and then load as a Numpy array.

you can try the following code:
dat_file = r"C:\Users\HPO2KOR\Desktop\Work\data1.dat"

with open(dat_file, 'r') as file:
    text = file.read()
    print(text)


How the embedding layer is trained in Keras Embedding layer How is the embedding layer trained in Keras Embedding layer?
(say using tensorflow backend, meaning is it similar to word2vec, glove or fasttext)

Assume we do not use a pretrained embedding.

Both the answers are wrong.
An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (for example 26 in the case of lower case alphabet) and on the other axis the dimensionality of your embedding space. The role of the embedding layer is to map a category into a dense space in a way that is useful for the task at hand, at least in a supervised task. This usually means there is some semantic value in the embedding vectors and categories that are close in this space will be close in meaning for the task.
This is related to one-hot encoding in a sense that it maps a discrete category into a vector feature representation. You could still do this for a neural network but if you use this in a dense layer you would create an enormous amount of weights of which most of them are not used regularly. Putting an embedding layer in between reduces the amount of learnable weights before feeding them to interact with other parts of your input. Another advantage is that the embedding matrix basically works as a lookup table, so you can really use the sparsity of the index of your category to look up what the current value of the embedding is and when applying backpropagating only adapting that entry of the weight matrix.
This blog post explains clearly about How the embedding layer is trained in Keras Embedding layer. Hope this helps.

Embedding layer uses embedding matrix for mapping data and is never updated during training.
There is no trainable parameters in the Keras Embedding layer. You can refer the Keras embedding layer docs for detailed understanding.


Choosing regularization method in neural networks When training neural networks, there are at least 4 ways to regularize the network:

 L1 Regularization
 L2 Regularization
 Dropout
 Batch Normalization

plus of course other things like weight sharing and reducing the number of connections, which might not be regularization in the strictest sense.

But how would one choose which of those regularization methods to use? Is there a more principled way than "just try everything and see what works"?

There are not any strong, well-documented principles to help you decide between types of regularisation in neural networks. You can even combine regularisation techniques, you don't have to choose just one.

A workable approach can be based on experience, and following literature and other people's results to see what gave good results in different problem domains. Bearing this in mind, dropout has proved very successful for a broad range of problems, and you can probably consider it a good first choice almost regardless of what you are attempting.

Also sometimes just picking a option you are familiar with can help - working with techniques you understand and have experience with may get you better results than trying a whole grab bag of different options where you are not sure what order of magnitude to try for a parameter. A key issue is that the techniques can interplay with other network parameters - for instance, you may want to increase size of layers with dropout depending on the dropout percentage.

Finally, it may not matter hugely which regularisation techniques you are using, just that you understand your problem and model well enough to spot when it is overfitting and could do with more regularisation. Or vice-versa, spot when it is underfitting and that you should scale back the regularisation.

Look at those algorithmic choices as additional hyperparameters and optimize them the same way as you do for your other hyperparameters. Typically this will require more data though.


Clustering Crime Data which has {latitute, longitude, crime-type} tuples I have a data set which has thousands of rows of {latitute, longitude, crime-type} tuples.

Sample Data:

41.757366519   -87.642992854   THEFT
41.910469677   -87.585822373   ROBBERY
41.751270452   -87.690708662   BURGLARY
41.757366519   -87.642992854   THEFT
41.757366519   -87.642992854   THEFT
..             ..              ..
..             ..              ..

I am trying to cluster these based upon the crime types.

For example, if in any  region, THEFT has a high frequency of occurrence, based on the data set, it should show up as a cluster. I have tried clustering using the lat-long data only, and that does not seem to have any meaning for this crime dataset.

I am fairly new to data mining, and gradually figuring my way out.

How can I cluster the data using the latitude and longitude values based such that the clusters are related to each other through the crime-type? Is there any tool available that can use the lat-long data and cluster them on the crime-type basis? Otherwise, I can even write a script once I understand how this can be done.

Also, has anyone had any previous experience in crime-data-mining? In what other ways can I find interesting patterns from a crime data-set?

Thanks a lot!

There is no meaningful way to combine "type" with a distance in meters.

My suggestion is that you:

split the data set by type.
cluster each type, with DBSCAN, haversine distance, and the same minpts/eps values for each crime type
compare the resulting clusters for similarity and differences

Separate the training data according to every possible value of crime type and form chunks and after that apply K-mean clustering for longitude and latitude on each separate chunk.

Ex, if possible crime type is "murder" and "burglary".

Then separate data in which "murder" is crime type and "burglary" is crime type.

Now you have two chunk of data. So apply K-Mean clustering on each chunk.


How to Deploy a Scikit-learn Model into the Cloud? I have generated a common Sklearn model for text classification which I want to make accessible in the cloud (there is no provider preference) as an API.

So far the closest solution that I managed to find is in this tutorial but it seems quite complex (getting the venv.zip dependencies package at the beginning is unclear, for instance) and specific (in my case NLTK and an external Stanford segmenter is involved for preprocessing and I cannot figure out where to put all these modules and how to invoke them).

Is there a decent and robust way to solve this challenge?

If you want to deploy a scikit-learn model to the cloud and be able to access it through an API, I am guessing that you will want to access the predict method.

The API

To build an API with python, the easiest is to use Flask or Django. I prefer Flask for these types of tasks because it has a simpler interface. Django offers you more lower-level control however.

You will have to create some route that the client will call by sending you the input data, and once you receive it on the server side, you'll just have to pass it through your Sklearn model, and send back the output to the client.

Run sklearn in the cloud

The other thing you need to take care of, is run Python on whatever cloud solution you'll use. In order to deploy your Flask/Django app, I would probably use Docker as it's well supported on many platform. Here is an example on how to deploy a Docker container on Azure

Flask is a python based package that is used to develop restful api services (api services).
when you run your application (directly from a command window), you will get a message telling you that you should not use this app as its in production environment. running the app from a cmd and using it is very risky and not reliable. so how to use an application (web service) that developed using flask?

The answer is:
you can use the IIS to host a flask application, but how to do this?

1- Install IIS: You’ll need to make sure that IIS is installed and configured with the CGI role service (this enables FastCGI as well).

2- lunch the IIS, make sure to run it as an administrator.
- click yes for starting the Microsoft Web Platform

if you do not get this message, you need to install the Platform Installer
Search for WFastCGI
Select the appropriate Python version (3.4 or 2.7.9). Click “Add”, then click “Install”.
if you have problem in finding the WFastCGI, you need to download it from this page wfastcgi, you can run the following command:
pip install wfastcgi

3- copy the wfastcgi.py from C:\Python34 (may be named C:\Python34_x86 if you
had an existing Python34 directory) to your Flask application root.

4- Double click “Handler Mappings”  (in the IIS)

5- Click “Add Module Mapping”

6- Click “Request Restrictions”. Make sure “Invoke handler only if request is mapped to:” checkbox is unchecked.

7- Go to the root server settings and click “FastCGI Settings”.

8- Double click it, then click the “…” for the Environment Variables collection to launch the Environment Variables Collection Editor.

9- Set the PYTHONPATH variable:
- Name: PYTHONPATH,    Value: your site directory

10-  set the WSGI_HANDLER (my Flask app is named app.py so the value is app.app - if yours is named site.py it would be site.app or similar)

11- that set.

the credit is for this page, you will find more details.
Deploying Flask on IIS