Machine Learning vs Deep Learning I am a bit confused by the difference between the terms "Machine Learning" and "Deep Learning". I have Googled it and read many articles, but it is still not very clear to me.

A known definition of Machine Learning by Tom Mitchell is:

  A computer program is said to learn from experience E with respect to
  some class of tasks T and performance measure P, if its performance at
  tasks in T, as measured by P, improves with experience E.

If I take an image classification problem of classifying dogs and cats as my taks T, from this definition I understand that if I would give a ML algorithm a bunch of images of dogs and cats (experience E), the ML algorithm could learn how to distinguish a new image as being either a dog or cat (provided the performance measure P is well defined).

Then comes Deep Learning. I understand that Deep Learning is part of Machine Learning, and that the above definition holds. The performance at task T improves with experience E. All fine till now.

This blog  states that there is a difference between Machine Learning and Deep Learning. The difference according to Adil is that in (Traditional) Machine Learning the features have to be hand-crafted, whereas in Deep Learning the features are learned. The following figures clarify his statement.

I am confused by the fact that in (Traditional) Machine Learning the features have to be hand-crafted. From the above definition by Tom Mitchell, I would think that these features would be learned from experience E and performance P. What could otherwise be learned in Machine Learning?

In Deep Learning I understand that from experience you learn the features and how they relate to each other to improve the performance. Could I conclude that in Machine Learning features have to be hand-crafted and what is learned is the combination of features? Or am I missing something else?

As a research area, Deep Learning is really just a sub-field of Machine Learning as Machine Learning is a sub-field of Artificial Intelligence.

1) Unsupervised Feature Learning

Conceptually, the first main difference between "traditional" (or "shallow") Machine Learning and Deep Learning is Unsupervised Feature Learning.

As you already know, successfully training a "traditional" Machine Learning model (ex: SVM, XGBoost...) is only possible after suitable pre-processing and judicious feature extraction to select meaningful information from the data.
That is, good feature vectors contain features distinctive between data points with different labels and consistent among data points with the same label.
Feature Engineering is thus the process of manual feature selection from experts. This is a very important but tedious taks to perform!

Unsupervised Feature Learning is a process where the model itself selects features automatically through training.
The topology of a Neural Network organized in layers connected to each other have the nice property of mapping a low-level representation of the data to a higher-level representation. Through training, the network can thus "decide" what part of the data matters and what part of the data doesn't. This is particularly interesting in Computer Vision or Natural Language Processing where it is quite hard to manually select or engineer robust features.

(picture credits: Tony Beltramelli)

As an example, let's assume we want to classify cat pictures. Using a Deep Neural Net, we can feed in the raw pixel values that will be mapped to a set of weights by the first layer, then these weights will be mapped to other weights by the second layer, until the last layer allows some weights to be mapped to numbers representing your problem. (ex: in this case the probability of the picture containing a cat)

Even though Deep Neural Networks can perform Unsupervised Feature Learning, it doesn't prevent you from doing Feature Engineering yourself to better represent your problem. Unsupervised Feature Learning, Feature Extraction, and Feature Engineering are not mutually exclusive!

Sources:

http://deeplearning.stanford.edu/tutorial/
https://arxiv.org/abs/1404.7828
https://arxiv.org/abs/1512.05616 (Chapter 2, Section 2)

2) Linear Separability

Deep Neural Networks can solve some non-linearly separable problems by bending the feature space such that features become linearly separable.
Once again, this is possible thanks to the network topology organized in layers mapping inputs to new representations of the data.

(picture credits: Christopher Olah)

Sources:
http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/

3) Statistical Invariance

Lastly, Deep Neural Networks are surpassing traditional Machine Learning algorithms in some domains because some architectures are showcasing Statistical Invariance (ex: Spacial Statistical Invariance with Convolutional Neural Networks and Temporal Statistical Invariance with Recurrent Neural Networks)

Check this Udacity video for more details:
https://www.youtube.com/watch?v=5PH2Vot-tD4

Okay, think of it like this. In machine learning algirithms, such as linear regression or random forest you give the algorithms a set of features and the target and then it tries to minimize the cost function, so no it doesnt learn any new features, it just learns the weights. Now when you come to deep learning, you have atleast one, (almost always more) hidden layer with a set number of units, these are the features that are being talked about. So a deep learning algorithm doesnt just learn the sets of weights, in that process it also learns the values for hidden units which are complex high level features of the trivial data that you have given. Hence while practicing vanilla machine learning a lot of expertise lies in your ability to engineer features because the algorithm isnt learning any by itself. I hope I answered your question.


How to find whether a dataset is blanced or imbalanced? I have few dataset to experiment classification(Multi-class). These datasets are about 400GB. I wanted to know whether the dataset is balanced or imbalanced. How to know that dataset is balance or imbalanced using any scientific way?

Typically, the representation of each class in a multi-classification problem should be equal. Say if there are 4 classes, then the ratio of count of samples in each class should ideally be n:n:n:n, most classification data sets do not have exactly same number of sample count in each class, which is fine and a lit bit of difference often does not matter. But if the difference is huge, say for example 100:5:9:13 then it matters and it is an imbalanced dataset.

coming to 400 GB of data to read - Depending on the type of your file, you can read it in chunks and then read and save the target variable( the one which has multi class labels) in another variable.

You can visualize this variable (containing target) using a bar chart which will show you the count of variables for each class. Along with that you can also calculate the distribution of your classes to get better understanding of data.

You can look at the number of samples for each class. Ideally, they all should be of equal proportion. If the sample of one class is considerably high than the rest, then the model will learn to predict that class more often than others and hence leading to overfitting.


Scientific explanation of Skip Connections I am searching for scientific work on skip-connections.

Everybody is talking about improving gradient flow through the Network, which feels like it makes sense. But I would love to understand and read what that means when it comes down to the math.

Other than the resnet paper I haven't found anything in depth about those magical skip-connections.

I am looking for any hints, papers or equations.

I recently read this paper which proposed a new technique without using skip-connections.It is proposed that the same accuracy can be achieved by increasing the width of the network rather than depth.


computational time and precision trade off I know that there are a number of predictive models (generized linear ones, trees, neural network, support vector machines, knn, Naive Bayes, ...) that have been proposed to perform various analytical tasks. Now I am striving to find appropriate references about their performance when the data becames "Big". In other words, how is their performance when the data becames really big. Does the training time increase more than linear? Is there any comparative benchmark between computational time and precision when the data becames high (for the various predictive models).

A good reference can be found there http://scikit-learn.org/stable/modules/


State of the art approaches for Information retrieval tasks based on deep learning I am interested in the state of the art approaches for information retrieval (IR) tasks, where you have a single query and a set of documents and the IR model will give you the best matched document.

I have worked on vector space models (tfidf-cosine similarirty ) and LSA.

I have also tried Wordnet, NER, fuzzy matching etc for improving the accuracy.

Now I would like to know how to improve accuracy of IR tasks by applying nueral networks, word embeddingstopic models etc by capturing more context/sematics information

Deep learning extends the ability of information retrieval (IR) systems.

Deep learning has proved to be a powerful tool for feature engineering. This improvement in feature engineering, over manual construction, has improved the quality of any machine learning system, including information retrieval.

Specifically, word embeddings create a dense, vectorized representation of words that encodes the semantic relationships based on co-occurrence. Both documents and queries can be projected into the same latent space and nearest-neighbors can be found. Word embeddings can be used to extend the posting index to by finding related terms. They can also be used for query-expansion / query rewriting.


How to predict the probability of an event? I have a dataset where a set of people donated for charity along with the dates of the donation. I have to find the probability of each donor donating in the next three months.

Data is available from August 2014 - February 2016. I have to predict the probability of each person donating for March-June 2016.

Any help would be appreciated?

Below is a snapshot of the data

id  date    amount
1   13-08-14    2485
1   21-11-14    2105
1   17-09-15    1359
2   13-08-14    2542
2   20-04-15    1276
2   12-10-15    2694
3   20-11-14    3556
4   28-07-15    3383
5   13-08-14    1698
5   11-12-14    1725
5   09-06-15    1376
5   17-09-15    3230

Regards

Please double-check if there's the only data you have got, because all you have is a single predictor date.

If this is indeed your only data source, then you only have a single predictor, and your independent variable is continuous. Now, you should plot date vs amount and fit a single linear regression. Does the fitting look good? Only you can tell because we don't have the full data-set.

If it's not a good fit, look at the plot and ask yourself does this look like a curve? If so, you might want to fit a spline curve or something like that.

You should also check the autocorrelation. This makes sense because your data look like a time series (you'll need to check it yourself). If this is the case, you might want to consider MA and ARCH model.

It's not possible for us to give you accurate advice because we don't know your data.

First you need to find in what kind of distribution is your data: linear, exponential, normal ... Etc.
After that you need to find the area size under the equation in which your event will occur.
It's all come down to what kind of distribution you are on.


What knowledge should I gain for developing a supervised image processing software that learns how to edit photos based on past behavior? I have done several machine learning projects but all of them have been connected to the traditional machine learning (predictions, classifications, etc.). I have currently been offered a project to finish in less than 6 months.

The idea is to develop/improve a pre-existing software. The software takes the image of a molecule from an advanced molecule and then tries to highlight the cell line with red, some times the software takes the background or the lines of other cells also as the highlighted part, and thus the user has to manually edit and trim such mistakes. The idea is to make the software learn from user's edits and behavior over time.

One thing I want to know is whether such a 6-month project is realistic for someone with no background in image processing and pattern recognition? Or is it going to be terribly difficult because I only have had experience with "data-oriented"/statistical machine learning?

My other question is; what type of concepts/topics should I dig deep into to learn the fundamentals of carrying out this project?

From the description of your problem, you need both Computer Vision and Deep Learning for a task like that. It is going to be extremely difficult, but you are more in advantage than anyone else, given you have a strong statistical and machine learning background. You don't have to worry a lot about the image processing part as there are libraries that will do that for you. You can look into PIL for that.The hard part is the learning from the edits part.A simpler way to solve this problem would be to focus on image processing and pick out the cell line clearly. The other way would be to train a Convnet on a large collection of 'labelled-images' of the cell-line, so that it is able to identify it in any picture. I do think it is quite a hard problem to solve but do give it a try. Cheers. All the best.


Can we take of benefit of using transfer learning while training a word2vec models? I am looking to find a pre-trained weights of an already trained models like Google News data etc. I found it hard to train a new model with enough amount (10 GB etc) of data for myself. So, I want to take benefit from transfer learning in which I would able to get pre-trained layer weights and retrain those weights on my domain specific words. So, definitely it will take relatively less time in training. Any sort of help will be highly appreciated.
Thanks in advance :)

Yes, you can take benefit of pre-trained models. Most famous one being the GoogleNewsData trained model which you can find here.

Pre-trained word and phrase vectors https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing

You can then load the vectors in binary format in your model using gensim as shown below.

>>> model = Word2Vec.load_word2vec_format('/tmp/vectors.txt', binary=False)  # C text format
>>> model = Word2Vec.load_word2vec_format('/tmp/vectors.bin', binary=True)  # C binary format

Here is a different pre-built model for English Wikipedia:

https://github.com/idio/wiki2vec/raw/master/torrents/enwiki-gensim-word2vec-1000-nostem-10cbow.torrent

Source: https://github.com/idio/wiki2vec/

Using a prebuilt model

Get python 2.7
Install gensim: pip install gensim
uncompress downloaded model: tar -xvf model.tar.gz
Load model in gensim:
from gensim.models import Word2Vec
model = Word2Vec.load("path/to/word2vec/en.model")
model.similarity('woman', 'man')

You can also use Stanford NLP Glove

http://nlp.stanford.edu/projects/glove/
https://github.com/stanfordnlp/GloVe

Here is a great compilation of pre-trained word2vec models.

https://github.com/3Top/word2vec-api#where-to-get-a-pretrained-models

Some additional pre-trained models:

http://www.zuccon.net/ntlm.html
https://code.google.com/archive/p/word2vec/

More on gensim and code here:  https://radimrehurek.com/gensim/models/word2vec.html

Quora forum with a similar questions

https://www.quora.com/Where-can-I-find-pre-trained-models-of-word2vec-and-sentence2vec-trained-on-Wikipedia-or-other-large-text-corpus
https://www.quora.com/Where-can-I-find-some-pre-trained-word-vectors-for-natural-language-processing-understanding

from gensim.models import Word2Vec
# Word2Vec is full model which is trainable but takes larger memory

from gensim.models import KeyedVectors
# KeyedVectors is reduced vector model which is NOT trainable but takes less memory

model = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True) #load pretrained google w2v

sen1 = 'w1 w2 w3'
sen2 = 'word1 word2 word3'
sentences = [[word for word in sen1.split()],[word for word in sen2.split()]]
total_examples = model_2.corpus_count

model_2 = Word2Vec(size=300, min_count=1) #initiate a full model
model_2.build_vocab(sentences) #add words in training dataset

#load words from pretrained google dataset
model_2.build_vocab([list(model.vocab.keys())], update=True)
model_2.intersect_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True, lockf=1.0)

#retrain pretrained w2v from new dataset
model_2.train(sentences, total_examples=total_examples, epochs=model_2.iter)


Found input variables with inconsistent numbers of samples I would appreciate if you could let me know how to resolve this error:
Code:
X = np.array(pd.read_csv('my_X_table1-1c.csv',header=None).values)
y = np.array(pd.read_csv('my_y_table1-1c.csv',header=None).values.ravel())
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=7)

def Ridgecv(alpha):
    return cross_val_score(Ridge(alpha=float(alpha), random_state=2),
                           X_train, y_train, 'mae', cv=5).mean()

The error is related to X_train, y_train:
ValueError: Found input variables with inconsistent numbers of samples: [1052, 1052, 3]

It seems that I missed the word "scoring". In fact, the extra 3 was related to the number of characters of 'mae'.

def Ridgecv(alpha):
    return cross_val_score(Ridge(alpha=float(alpha), random_state=2),
                           X_train, y_train, scoring='mae', cv=5).mean()

It should be in sequence:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test=train_test_split(X,Y,random_state=101,test_size=0.3)

and then it should be in fit method(x_train,y train)


Is NLP suitable for my legal contract parsing problem? My company has a product that involves the extraction of a variety of fields from legal contract PDFs. The current approach is very time consuming and messy, and I am exploring if NLP is a suitable alternative.
The PDFs that need to be parsed usually follow one of a number of "templates". Within a template, almost all of the documents are the same, except for 20 or so specific fields we are trying to extract. That being said, there are sometimes slight inconsistencies such as missing/added spaces, punctation, etc. Currently there are about 15 templates, but sometimes another gets added. Most of the data that needs to be extracted is in tabular form, but a few values are littered in paragraph text.
With the current, non-ML approach, we use a proprietary PDF parser that extracts out all of the tables into a json format. This parser must be custom configured for every new PDF template that appears, and this is time consuming. After converting to json, a custom regex expression must be made for each value in each template, leading to several hundred regexes total. Usually this approach is pretty accurate, but sometimes the slight inconsistencies can break it, meaning a the regex needs to be revised.
I'm wondering if NLP Named Entity Recognition would be a cleaner solution to this problem. My idea is to label all of the values in a few hundred sample documents and then train a custom NER model in a library like Spacy or Flair. Ideally, we could feed in the raw, extracted text from the PDFs instead of having to configure the custom parser to extract json.
The advantage I see in using the NLP approach is that we wouldn't have to configure the custom parser and write a bunch of regexes every time a new template appears. At worst, we would have to label a few of the new documents every time a template is added, which would presumably be faster and easier than the current approach. I think we could also generate tons of synthetic training data easily by swapping labeled values between different documents.
I am concerned that, using an ML approach, we wouldn't be able to achieve near perfect accuracy, which is a requirement. I'm also not sure how well NLP can perform on raw text from tables as opposed to paragraphs. The nice thing is that the documents tend to be very similar within templates.
I've never done NLP before, so I'm wondering if anyone here thinks this approach would be worthwhile exploring. If this is feasible, does anyone have suggestions on how to get the best results?

Here's what I would have done.

extract raw data from the the pdfs.
use rule based NER from spacy(it's quite fast), but it will require some manual pattern making.
Also apply multiprocessing if you can.
https://spacy.io/usage/rule-based-matching
You can also write a document classification to segregate the templates and send the templates to appropriate code,running parallely.
Also checkout blackstone library for legal texts, it might be of some use https://spacy.io/universe/project/blackstone

A couple of pictures of the documents with your target data highlighted might help. Also your approach seems sound enough.
I agree you wouldn't get perfect results each time, however there is a concept of Human In the Loop. Where the data extracted by your program would have to be verified by a human once before certifying. This is basically a digital signature of the person verifying the data to have an audit trail. From what I understand, you're trying to extract just a few entities from the documents. So glancing over the data to verify whether things indeed make sense shouldn't be too difficult.
This approach would certainly work faster than editing the template and making changes to the regular expressions.
Alternatively, your problem could also be construed as contextual summarization where the summary would be an extract of your NERs. I'm not sure how effective this would be (maybe someone more experienced could chime in) but you could go the deep learning route and train an attention based model like BERT on your custom dataset containing the extracted NERs as your target summarization.


PySpark v Pandas Dataframe Memory Issue Suppose I have a csv file with 20k rows, which I import into Pandas dataframe. I then run models like Random Forest or Logistic Regression from sklearn package and it runs fine. However, when I import into PySpark dataframe format and run the same models (Random Forest or Logistic Regression) from PySpark packages, I get a memory error and I have to reduce the size of the csv down to say 3-4k rows. Why does this happen? Is this a conceptual problem or am I coding it wrong somewhere?

For Pandas dataframe, my sample code is something like this:

df=pd.read_csv("xx.csv")

features=TfIdf().fit(df['text'])
....
RandomForest.fit(features,labels)

And for PySpark, I'm first reading the file like this:

data = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true')\
.load('xx.csv')
data.show()
from pyspark.ml.feature import RegexTokenizer, StopWordsRemover, CountVectorizer
from pyspark.ml.classification import LogisticRegression
# regular expression tokenizer
regexTokenizer = RegexTokenizer(inputCol="converted_text", outputCol="words", pattern="\\W")
# stop words
add_stopwords = ["http","https","amp","rt","t","c","the"]
stopwordsRemover = StopWordsRemover(inputCol="words", outputCol="filtered").setStopWords(add_stopwords)
# bag of words count
countVectors = CountVectorizer(inputCol="filtered", outputCol="features", vocabSize=10000, minDF=5)
from pyspark.ml import Pipeline
from pyspark.ml.feature import OneHotEncoder, StringIndexer, VectorAssembler
label_stringIdx = StringIndexer(inputCol = "Complaint-Status", outputCol = "label")
pipeline = Pipeline(stages=[regexTokenizer, stopwordsRemover, countVectors, label_stringIdx])
# Fit the pipeline to training documents.
pipelineFit = pipeline.fit(data)
dataset = pipelineFit.transform(data)
dataset.show(5)
(trainingData, testData) = dataset.randomSplit([0.7, 0.3], seed = 100)
print("Training Dataset Count: " + str(trainingData.count()))
print("Test Dataset Count: " + str(testData.count()))
from pyspark.ml.classification import RandomForestClassifier
rf = RandomForestClassifier(labelCol="label", \
                            featuresCol="features", \
                            numTrees = 100, \
                            maxDepth = 4, \
                            maxBins = 32)
# Train model with Training Data
rfModel = rf.fit(trainingData)
predictions = rfModel.transform(testData)
predictions.filter(predictions['prediction'] == 0) \
    .select("converted_text","Complaint-Status","probability","label","prediction") \
    .orderBy("probability", ascending=False) \
    .show(n = 10, truncate = 30)

I was trying for lightgbm, only changing the .fit() part:

from mmlspark import LightGBMClassifier
lgb = LightGBMClassifier(learningRate=0.3,
                           numIterations=100,
                           numLeaves=31)
lgb_model=lgb.fit(trainingData)
predictions = lgb_model.transform(testData)
predictions.filter(predictions['prediction'] == 0) \
    .select("converted_text","Complaint-Status","probability","label","prediction") \
    .orderBy("probability", ascending=False) \
    .show(n = 10, truncate = 30)
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
evaluator = MulticlassClassificationEvaluator(predictionCol="prediction")
evaluator.evaluate(predictions)

And the dataset has hardly 5k rows inside the csv files. Why is it happening? How can I solve it?

While I can't tell you why Spark is so slow (it does come with overheads, and it only makes sense to use Spark when you have 20+ nodes in a big cluster and data that does not fit into RAM of a single PC - unless you use distributed processing, the overheads will cause such problems. For example, your program first has to copy all the data into Spark, so it will need at least twice as much memory. Probably even three copies: your original data, the pyspark copy, and then the Spark copy in the JVM. In the worst case, the data is transformed into a dense format when doing so, at which point you may easily waste 100x as much memory because of storing all the zeros).

Use an appropriate - smaller - vocabulary.

There is no use in including every single word, as most of them will never score well in the decision trees anyway!

It's safe to assume that you can omit both very frequent (stop-) words, as well as rare words (using them would be overfitting anyway!). So use min_df=10 and max_df=1000 or so.

Pandas dataframes can be rather fickle. I've observed code running fine until one line somewhere tries to load more data in memory than it can handle and it all breaks apart, landing a memory error. Even if the rows are limited, the number of columns and the content of each cell also matters. If it's all long strings, the data can be more than pandas can handle. Now, if you train using fit on all of that data, it might not fit in the memory at once. Look here for one previous answer.

Pyspark, on the other hand, has been optimized for handling 'big data'. An rdd contains many partitions, which may be distributed and it can spill files to disk. Here, you can read more on it.


What does it mean that classes are mutually exlcusive but soft-labels are accepeted? The Tensorflow's documentation of softmax_cross_entropy_with_logits:

  Measures the probability error in discrete classification tasks in
  which the classes are mutually exclusive (each entry is in exactly one
  class). For example, each CIFAR-10 image is labeled with one and only
  one label: an image can be a dog or a truck, but not both.

  NOTE: While the classes are mutually exclusive, their probabilities
  need not be. All that is required is that each row of labels is a
  valid probability distribution. If they are not, the computation of
  the gradient will be incorrect.

At first glance it may seen contradictory, but my guess is that, provided these conditions for the classes and for the labels, the probability distribution of the labels have always one maximum. For instance, labels = [0.5 0.5] is a valid probability distribution in a binary case but it does not comply with the exclusivity of the classes. If I'm wrong, I don't get the idea behind the documentation.

It is also pointed out that for mutually elusive probabilities, sparse_categorical_cross_entropy should be used, but I think that softmax_cross_entropy_with_logits can be used if the labels are one-hot encoded, which is a valid probability distribution (a deterministic one)

Edit:

When going around this question again, I'm wondering if in a multi-class exclusive case where the only constraint on the labels are that they have to be a valid probability distribution, labels = [0.5 0.5] should be a valid instance label. This label means that the annotator nor the net can tell if this ground-truth instance belongs to class_0 or class_1...

What does it mean that classes are mutually exclusive but soft-labels are accepted?

As it can be seen from here, tf.nn.softmax produces just the result of applying the softmax function to an input tensor. The softmax "squishes" the inputs so that sum(input) = 1; it's a way of normalizing. The shape of the output of a softmax is the same as the input - it just normalizes the values. The outputs of softmax can be interpreted as probabilities. In contrast, tf.nn.softmax_cross_entropy_with_logits computes the cross-entropy of the result after applying the softmax function (but it does it all together in a more mathematically careful way). It's mathematically careful due to the fact that $y_i$ in $log(y_i)$ can be zero. As you can read from here, A randomly-initialized softmax layer is extremely unlikely to output an exact 0 in any class. But it is possible, so worth allowing for it. First, don't evaluate log(yi) for any y′i=0, because the negative classes always contribute 0 to the error. Second, in the practical code you can limit the value to something like $log( max( y_predict, 1e-15 ) )$ for numerical stability - in many cases, it is not required, but this is sensible defensive programming. I encourage you to take a look at the answers to this question.

  NOTE: While the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of labels is a valid probability distribution. If they are not, the computation of the gradient will be incorrect.

The first sentence means that your classifier may not be able to classify the labels exactly as they are, one-hot-encoded. What it does is to find the chance that how likely it is that the input belongs to each class. And this won't make a problem if it does not have mutually exclusive output vector as the algorithm implies. It just needs a vector that the sum of its entries is equal to one. If they are not, the computation of the gradient will be incorrect. I guess this line is added to announce that the outputs of this differentiable component will not be one-hot-encoded and it's because of the nature of these nets. The first layers of convolutional networks are like bases vectors and each instance of classes share these bases. All inputs are made up of these bases.

  I'm wondering if, in a multi-class exclusive case where the only constraint on the labels is that they have to be a valid probability distribution, labels = [0.5 0.5] should be a valid instance label. This label means that the annotator nor the net can tell if this ground-truth instance belongs to class_0 or class_1...

Basically, for multi-label classification, your input may have different labels. Consequently, your classes won't be mutually exclusive anymore. Moreover, in those cases, we don't use softmax as the last layer. You have to have sigmoid for each output and your cross-entropy cost function will be slightly different 1. Consequently, the output of each entry should be a valid probability, that is why sigmoid is used, and the label vectors for such tasks is a not one-hot-encoded anymore. different classes have different entries and based on the existence of instances of each class, the corresponding entry should be one.


How to find the relationship between two categorical variables? I'm building a model and trying to get the relationship between two multi-level categorical variables.

For example, I want to know the relationship between race and likelihood of graduate from collage, we have 5 races and YES or NO for graduation. How to find the which race have higher likelihood or which have lower?

Thanks

Traditional statistics like Chi-squared tests and Cramer's V can be used to determine relationship between two categorical features.

One suggestion is to label encoding on these categorical variables and find the correlation.
Or I believe we can use count plot for the same.


Monthly data versus yearly prediction How would you see the relation of a monthly variable categorical with a yearly target value float for differents years in order to predict.

In the farming industry the relation is not so simple for categorical values as the drought level , like pre-harvest period has more weight for some months on the yearly result because some months are more relevant.

What kind of null hypothesis test would you do for with both variables to conclude
I was thinking of anova
Any other suggestions.

You could consider each month as an explanatory variable. And the year as response variable. Because, for each year, you'll have 12 monthly variables.

With a model, you could find the relationship that has each month value with the final output. For that, you can perform a linear model (your categorical variables will or must be recoded as dummy variables for that estimation).

If you want to check the effect of the variable by itself. Yearly Target = a * Monthly Variable. I agree with @Leevo.

You can use linear or Loess interpolation. Alternatively you can take monthly averages in case aggregation is not a problem for your task.


Keras Embedding Layer main paper I need to explain the word embedding layer of Keras in my paper, mathematically. I know that keras initialize the embedding vectors randomly and then update the parameters using the optimizer specified by the programmer. I want to explain my architecture in an academic paper, therefore I need to explain each layer with a formal formula.
Is there a paper that explains the method in details to reference it? or can you please guide me to how to compose the method formally?

Thanks a lot

I think I found the answer. The embedding layer in keras is nothing more than a set of vectors for distinct words. The keras embedding layer initializes the word embedding with some random values (the default values from a uniform distribution)and then updates the values when train the whole network. Therefore, there is no need to compose the model in details, because the backpropagation do it perfectly :)


Few activation functions handling various problems - neural networks How can a few activation functions in neural networks handle so many different problems?

I know some basics theory behind ANN, but I can't get what functions like the sigmoid function etc. have in common with for example image classification?

Image classification and other task can be expressed as function approximations and, in theory, neural networks can approximate (almost) any function (given few assumptions on the activation function (see Universal Approx. Theorem)).

However, in practice, not all functions fulfilling these assumptions work equally well. Popular activation functions usually share some properties that allow neural networks to learn efficiently in practice, e.g. they are continuously differentiable (for gradient descent), close to the identity near zero (accelerates initial learning from small random weights) and so on.

Activation functions like the sigmoid function are not directly related to image classification or any other tasks. Rather, they allow for efficient training of neural networks, which, in turn, can represent a wide variety of tasks using different architectures and cost functions.

Two keys things which work in favour of sigmoid and tanh functions are:

They are bounded functions over their entire domain(which happens to
be the entire real line).
Their derivatives are easy to derive and computationally inexpensive.

Also, the purpose of using sigmoid and tanh are different. Their boundaries are (0,1) and (-1,1) respectively. So, sigmoid is good for calculating probability of an event and tanh is better for classifying two classes.

A basic problem in image classification is predicting whether an image has a cat in it or not. So, using tanh function we get a final value of which ranges be (-1,1). So, you can decide basis whether the final value is negative or positive to reach the final conclusion of image classification.


How to use Cross Entropy loss in pytorch for binary prediction? In the pytorch docs, it says for cross entropy loss:

  input has to be a Tensor of size (minibatch, C)

Does this mean that for binary (0,1) prediction, the input must be converted into an (N,2) tensor where the second dimension is equal to (1-p)?

So for instance if I predict 0.75 for a class with target 1 (true), would I have to stack two values (0.75; 0.25) on top of each other as input?

Actually there is no need for that. PyTorch has BCELoss which stands for Binary Cross Entropy Loss. Please check out original documentation here. Here is a quick example:

m = nn.Sigmoid() # initialize sigmoid layer
loss = nn.BCELoss() # initialize loss function
input = torch.randn(3, requires_grad=True) # give some random input
target = torch.empty(3).random_(2) # create some ground truth values
output = loss(m(input), target) # forward pass
output.backward() # backward pass

Your network must have two final neural nodes, if your input can belong to one of two different classes. The target tensor must then be of the shape (minibatch) and only contain zeros and ones.

Generally your network must output a Tensor with shape (minibatch, C), where C is the number of classes your data can be classified in. The target tensor must be of shape (minibatch) and consist of only numbers of type long, that are element of the set {0, ..., C-1}.

Maybe a real world example: Your network is fed with pictures of dogs, cats and pigs. Therefore it has 3 final nodes which (usually) represent the probability of the input image showing a dog (class 0), cat (class 1) or pig (class 2). Let's assume, you put 10 images into your network  at once (minibatch = 10). Then your target tensor could be:

torch.LongTensor([0, 2, 1, 0, 1, 0, 2, 2, 1, 0, 0, 1])

This can be interpreted as the first image of your batch shows a dog, the second image a pig, the third image a cat, ...


Data science which is not part of AI? I have understood that data science is part of  AI. However, do we have some area in data science which we can say does not fall under AI? If yes, please name them.

I would not concern yourself too much about any structuring of knowledge that declares that one subject is categorised as one thing or another. These structures are often wrong and knowledge in general is more fluid and difficult to define than can be viewed as some kind of Venn diagram.

In addition, both Data Science and AI are poorly defined, and have more in common with marketing terms used to recruit staff and sell products than stricter academic subject areas such as Maths or Modern Languages. You will find Data Scientists working on "AI" projects and "AI Engineers" using basic statistics to analyse data.

Having said that, I would prefer to say that Data Science and AI share a common set of tools, especially when it comes to machine learning. Neither contains the other in my opinion.

The popular definition of AI is that machine doing action which are similar as human applying intelligence.

Human being have limited search capability. So "search" should not be part of AI. However, it is definitely part of data science.


Training Keras model with multiple CSV files I'm currently trying to train a Keras model on several large CSV files. I can fit one in memory, but not all combined. From my point of view, there are several ways to deal with this problem.
I could merge all of the data into a single CSV file and then read the CSV by the chunks of the data. This would simplify the process, however, I would not be able to shuffle the data each epoch.
Other approach that I can think of is to create a custom fit_generator. However I'm not sure how to actually implement this. Should I create a special generator for each dataset, and then cycle them? This would allow me to shuffle each dataset every epoch and even their order.
I believe that it would be more elegant to implement just a single generator, and manage all of the files inside. However to make this work, I would need to know the number of batches prior to training. That would require me to first get number of samples in each file, which would take a while. Also, I would need to sort out the issue of some lines being incorrect after reading the file with pandas. That limits me from using some fast os-level functions to get the number of lines.
Or is it possible to actually create a fit_generator without setting the number of batches prior to training? What approach would you recommend?

I would use tensorflow 2.0 with tf.data
import tensorflow as tf

filenames = ["filename1", "filename2", ...]

dataset = tf.data.Dataset.list_files(filenames, seed=42, shuffle=True)

# this reads 5 text files at a time, skips the first row of each file
dataset.interleave(lambda filename: tf.data.TextLineDataset(filename).skip(1), cycle_length=5, num_parallel_calls=tf.data.experimental.AUTOTUNE)

for line in dataset.take(5):
    print(line)

You can then pass in the dataset objects directly into tensorflow.keras for training.
There are some good examples here but please beware that this is NOT for tensorflow 2.0, and somethings may have changed

This may be a potential option for you to consider.
Take a sub-sample of records from each csv file. This should be perfectly fine if your data is (at least somewhat) normally distributed.  If the distribution is not normal, normalize it or standardize it.
from sklearn import preprocessing
normalized_X = preprocessing.normalize(X)

from sklearn import preprocessing
standardized_X = preprocessing.scale(X)

Then,
# To get 3 random rows
# each time it gives 3 different rows
# df.sample(3) or
df.sample(n = 3)

# Fraction of rows
# here you get .50 % of the rows
df.sample(frac = 0.5)

Also, consider this.
# Split the data between the Training Data and Test Data
xTrain , xTest , yTrain , yTest = train_test_split(X , y ,
                                                  test_size = 0.30 ,
                                                  random_state = 0,
                                          ----->  stratify = y)


Cluster evolution over time I have a dataset of transactional data with customer ID and I want to segment the dataset into groups using cluster analysis. I'm interested in following the evolution of each cluster over time, but since customers have very different behaviours (roughly 50% of the time a customer will change cluster the week after), I was wondering what would be a statistically sound approach. Is it a good idea to train a clustering algorithm every week and look backwards at the weekly evolution of each segment?

You can try

Dynamic mode decomposition.
Dynamic Time Warping. Found a nice resource on Towards data science blog.

These two have proven better approaches than PCA for time series clustering.
Happy coding

Cluster once.

Study the clusters and refine them to define classes.

Then classify points to these classes.


Recurrent (CNN) model on EEG data I'm wondering how to interpret a recurrent architecture in an EEG context. Specifically I'm thinking of this as a Recurrent CNN (as opposed to architectures like LSTM), but maybe it applies to other types of recurrent networks as well

When I read about R-CNNs, they're usually explained in image classification contexts. Theyre typically described as "learning over time" or "including the effect of time-1 on the current input"

This interpretation/explanation gets really confusing when working with EEG data. An example of an R-CNN being used on EEG data can be found here

Imagine I have training examples each consisting of a 1x512 array. This array captures a voltage reading for 1 electrode at 512 consecutive time points. If I use this as input to a Recurrent CNN (using 1D convolutions), the recurrent part of the model isn't actually capturing "time", right? (as would be implied by the descriptions/explanations discussed earlier) Because in this context time is already captured by the second dimension of the array

So with a setup like this, what does the recurrent part of the network actually allow us to model that a regular CNN can't (if not time)?

It seems to me that recurrent just means doing a convolution, adding the result to the original input, and convolving again. This gets repeated for x number of recurrent steps. What advantage does this process actually give?

The recurrent part of a network allows you, generally speaking, to model long and short term dependencies. So your model can have some sense of state.

This is typically advantageous if you're using timeseries. For example, if you have data from a heart rate monitor and like to like to classify between resting, stress and recovering. If your datapoint says your heart rate is at 130, it depends on whether you are recovering from high loads or something else.

Edit:
I forgotten your second question.

  It seems to me that recurrent just means doing a convolution, adding the result to the original input, and convolving again. This gets repeated for x number of recurrent steps. What advantage does this process actually give?

I could think off a few possible answers. By convoluting the recurrent part you kind of filtering it. So you get a cleaner signal and errors won't stack as much. Vanilla rnn suffer from exploding vanishing gradients, so this could be his approach to overcome it.
Furthermore, you are embedding your features within the rcnn, which can lead, as he stated, to more paths to exploit. Which makes it less prone to overfitting, thus more generalizable.

Remember that CNNs are feature detectors. The output of a convolutional layer is a matrix that signals where certain feature was detected.

Therefore, recurrent CNNs are recurrent neural networks that learn sequences of features, where those features are also learnt during the training.


Selecting ML algorithm for music composition I'm a composer and programmer. I'd like to use ML for composing music. There's already research on the general problem of composing by machine in known musical styles. I'm more interested in the specific problem of using a computer to assist myself in composing... to speed up certain repetitive tasks, or to delight me with unexpected ideas.

First, I want to design ML that can make a good/bad categorization on a single short passage. (Not looking to generate music at first.)

It's pretty easier to decompose such a passage into significant features. I can say things like

maybe there are particular "intervals" (harmony) present, and others absent
maybe notes are spread out evenly from low to high, or maybe not
maybe notes are clustered around a certain register rather than spread out
maybe the rhythm is particularly uneven
maybe a particular note occurs a lot

I can easily write algorithms that produce a list of such features for any given passage, and I can even be pretty sure that I'm including all of the relevant features. For an "art form," music is surprisingly amenable to analysis.

The trick is, first of all, there will be hundreds of observations, and second, that no one of these observations is good or bad in itself. Say we have three factors A, B, and C. A is not bad alone, but perhaps when A and B occur together, that's bad. But then maybe if A, B, and C all occur together, it becomes good again! And then maybe a fourth factor D, changes the context so that A, B, and C no longer matter much.

This seems like an ML problem, but I wonder what the best approach is?

I can construct or compose hundreds of examples of music labeled good and bad, and identify hundreds of "features" in each example. Some of the features are "present/not present" categories, while others are numbers that can be scaled from 0 to 1.

Is there a general and simple learning algorithm that even an ML novice such as myself (but I'm an experienced programmer) can implement?

I have to generate my own training examples and would be limited to around 300 to 1000, as I'm training it on simple passages in my style. This may actually be good problem to have... my job is not that complex compared to general musical intelligence.

I will contrast this task with image recognition. An image has millions of pixels and the features are spread out, occurring in endless  permutations. It makes my head hurt to try to figure out what's going on in a deep neural net.

However, in music, there will be at most a few hundred features. The patterns will not occur in endless permutations. In fact, in stripped-down situations my wild guess is that you could make an 80% accurate classifier by looking at 10 features (assuming you have the weights and combinations well-trained). I'm aiming somewhat higher than that (I need more features and a bit more accurate).

To give you some idea of my data set: I'm in the early stages of writing the software to extract the features, but I have a pretty good mental idea of what I'm aiming for (been composing by hand for years and years). To make this more concrete I created a web page showing what the data will look like. This is very simple and just totally made up.

http://theunexpectedpearl.com/dataset.html

This shows four rows. Each row is an example passage of music that I will compose or generate automatically. Each row is classified "good" or "bad" by my own ear. The rest of the row is a long list of features, each of which is either "yes/no" or a single number scaled from 0.0 to 1.0.

I don't think a Naive Bayes classifier will work for this (if I understand correctly) because features are neither good nor bad in and of themselves... it is only combinations of them (context) that matter. In fact, the very same feature can contribute to "goodness" in one context, and be an immediate strong indication of "bad" in another context.

The data could possibly be structured differently. In practice, this spreadsheet would be "sparse" and maybe features should be grouped in some way, or arranged in a tree.

Let us formulate this problem in such a way that it can be understood from a machine learning perspective. You have a set of instances $X$ where each instance $x_i \in \mathbb{R}^m$ where $m$ is the dimensionality of the instance. In other words $m$ is the number of features that describe the instance. Your problem intends to go from a set of features to a class label good or bad. Thus, this is a mapping from $\mathbb{R}^m$ to $y \in \{0, 1\}$.

How to achieve this mapping?

This is when we will use the machine learning algorithms. We will train a model to effectively approximate the function which gives the output label from a set of inputs. It is evident that sparse features (low information entropy) will complicate the mapping function and will thus provide worse results. This is why feature engineering is of upmost importance for machine learning. It is probably the hardest part of the machine learning pipeline, however it is the lead factor in dictating your results.

You can use some feature reduction techniques in order to remove features which are uninformative with respect to the output label. Some techniques that I use frequently are principle component analysis (PCA), linear discriminant analysis (LDA). Alternatively, you can use some projection methods to reduce the dimensionality of the data whilst maintaining separation between the classes. Such techniques are Isomap, MDS, Spectral Embeddings and TSNE. You can check to see which is best suited for your type of data.

How to choose a model?

Firstly, your problem is a supervised classification problem. This already narrows the types of models you can use. Furthermore, model selection is based on some key factors such as: the number of instances you have, the number of features per instance and the number of output nodes. You should also keep in consideration that the separability of the probability distribution between the output classes will impact the performance of the model directly. For example discriminating between cars and oranges is much easier than oranges and clementines.

In your case, you have 1,000 instances and around 13 features. This means that deep learning based techniques are possible but discouraged. You do not have enough data. You can then attempt the following popular classification models

Support Vector Classifier
Naive Bayes
K-Nearest Neighbors
Decision Trees
Random Forests

To evaluate which model performs the best you will use the accuracy attained with a trained model on a test set. This set should be drawn independently from the training set as to catch overfitting. This is when the model cannot generalize to new data.

In code

Assuming matrix $X$ contains the data where rows are the instances and columns are the features, and matrix $Y$ contains the labels.

First we split our data into a training and testing set

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33)

from sklearn.svm import SVC
clf = SVC()
clf.fit(X_train, y_train)
print('Score: ', clf.score(X_test, y_test))

from sklearn.neighbors import KNeighborsClassifier
neigh = KNeighborsClassifier(n_neighbors=3)
neigh.fit(X_train, y_train)
print('Score: ', neigh.score(X_test, y_test))

from sklearn import tree
clf = tree.DecisionTreeClassifier()
clf.fit(X_train, y_train)
print('Score: ', clf.score(X_test, y_test))

from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators = 100)
forest.fit(X_train, y_train)
print('Score: ', forest.score(X_test, y_test))

This should be a starting point. Let us know if you fall into any problems, and let us know what accuracy you are getting we can then look deeper into these models and better suit them to your data source.

I'll offer up a few suggestions to get the creative juices flowing.

First, since you have two tasks, you could consider two models configured as a Generative Adversarial Network where

One model generates musical passages
One model classifies musical passages

The output from the generative model is then evaluated by the classifier.  The output from the classifier can then be used to improve (train) the generative model.

I imagine these would be trained on the same labeled data set.  I would use a Recurrent Neural Network to train the classifier and tune the snot out of it (so to speak).
I would use a Generative Neural Network for the generative model.

Maybe you were looking for more direction.  If so, sorry to disappoint.  I figured you may have been looking for better ways to search for help.

Best of luck.


Difference between train, test split before preprocessing and after preprocessing I am new to machine learning. I am bit confused in preprocessing. Generally,

Scenario-1: I am splitting the dataset into train,test and validation and applying the transformations like fit_transform on train and transform on test.

Scenario-2: The other method is applying transformations on the entire dataset first and then split the dataset into train,test and validation. I am bit confused in choosing , dividing the data before preprocessing and feature engineering or after preprocessing and feature engineering. Looking for a nice answer with effects and casues.

You should absolutely adopt the first scenario. That's because the transformers that you use have some parameters (e.g. mean and standard deviation in case of standard scalar) and this parameters are learned from data like the parameters of your machine learning model. As you know, you should not use the validation and test data for learning the model parameters and for the same reason, you should not use them for learning the transformer parameters. As a result, you should just use the training samples for fitting your transformer parameters if you want to try a practical machine learning scenario.

The prepossessing transformations have to be applied on all data sets ( train, test and validation). Whatever feature you add on one subset, has to be added in the other subsets.

If you're validation and test data are chosen randomly and you apply the exact transformations on all subsets both scenarios would work but the second scenario is better in some ways:

First it easier to code you'll just have to write the transformation lines once.
Second it is probably faster, you'll benefit from the power of numpy if you're using python for example
Third it is less risky, take the following example. You have a categorical variable and for some reason you decide to replace the missing values in that variable with the value with highest occurrence. If the value occurrences are close you could have different value in your subsets. The same could happen if you replace a numerical value with the mean. This is unlikely if you pick your subsets randomly but you know, there is no prefect randomness.

I would recommend choosing the second option as it is easier and less risky. But still, you can try both at some point and you'll see the advantages and disadvantages of each.


Pyspark coverting timestamps from UTC to many timezones This is using python with Spark 1.6.1 and dataframes.

I have timestamps in UTC that I want to convert to local time, but a given row could be in any of several timezones. I have an 'offset' value (or alternately, the local timezone abbreviation. I can adjust all the timestamps to a single zone or with a single offset easily enough, but I can't figure out how to make the adjustment dependent on the 'offset' or 'tz' column.

There appear to be 2 main ways of adjusting a timestamp:
using the 'INTERVAL' method, or using pyspark.sql.from_utc_timestamp.

Here's an example:

data = [
  ("2015-01-01 23:59:59", "2015-01-02 00:01:02", 1, 300,"MST"),
  ("2015-01-02 23:00:00", "2015-01-02 23:59:59", 2, 60, "EST"),
  ("2015-01-02 22:59:58", "2015-01-02 23:59:59", 3, 120,"EST"),
  ("2015-03-02 15:59:58", "2015-01-02 23:59:59", 4, 120,"PST"),
  ("2015-03-16 15:15:58", "2015-01-02 23:59:59", 5, 120,"PST"),
  ("2015-10-02 18:59:58", "2015-01-02 23:59:59", 4, 120,"PST"),
  ("2015-11-16 18:58:58", "2015-01-02 23:59:59", 5, 120,"PST"),
  ("2015-03-02 15:59:58", "2015-01-02 23:59:59", 4, 120,"MST"),
  ("2015-03-16 15:15:58", "2015-01-02 23:59:59", 5, 120,"MST"),
  ("2015-10-02 18:59:58", "2015-01-02 23:59:59", 4, 120,"MST"),
  ("2015-11-16 18:58:58", "2015-01-02 23:59:59", 5, 120,"MST"),
  ...
]

(I realize the offset and tz columns aren't consistent - this isn't real data)

df = sqlCtx.createDataFrame(data, ["start_time", "end_time", "id","offset","tz"])
from pyspark.sql import functions as F

these two options both do what is expected:

df.withColumn('testthis', F.from_utc_timestamp(df.start_time, "PST")).show()
df.withColumn('testThat', df.start_time.cast("timestamp") - F.expr("INTERVAL 50 MINUTES")).show()

But if I try to replace the "PST" string with df.tz, or the " 50 " string with df.offset.cast('string'), I get a type error:

  TypeError: 'Column' object is not callable

I've tried variations on this, but to no avail.

if the data type of your "tz" column is a string, then you can do:

df.select(from_utc_timestamp(df.start_time,tz).alias('start_time')).show()


XOR Neat Python Result is incorrect? I ran the Neat Python XOR example and a neural network it found has the following properties:

  Best individual in generation 22 meets fitness threshold - complexity:
  (2, 8) Number of evaluations: 3450

  Best genome: Nodes:
          NodeGene(id=0, type=INPUT, bias=0.0, response=4.924273, activation=sigmoid)
          NodeGene(id=1, type=INPUT, bias=0.0, response=4.924273, activation=sigmoid)
          NodeGene(id=2, type=OUTPUT, bias=0.0, response=4.921960812206318, activation=sigmoid)
          NodeGene(id=3, type=HIDDEN, bias=-2.340619398413843, response=4.930091408143276, activation=sigmoid)
          NodeGene(id=4, type=HIDDEN, bias=0.4674809078722945, response=4.918937894580367, activation=sigmoid) Connections:
          ConnectionGene(in=0, out=2, weight=1.0534262789427578, enabled=True, innov=0)
          ConnectionGene(in=1, out=2, weight=-1.6479324848766828, enabled=True, innov=1)
          ConnectionGene(in=0, out=3, weight=-3.5479647367966765, enabled=True, innov=2)
          ConnectionGene(in=3, out=2, weight=5.970619032226201, enabled=True, innov=3)
          ConnectionGene(in=1, out=3, weight=4.634338591600734, enabled=True, innov=4)
          ConnectionGene(in=0, out=4, weight=0.19203601809123594, enabled=True, innov=5)
          ConnectionGene(in=4, out=2, weight=-0.6588709650159562, enabled=True, innov=6)
          ConnectionGene(in=4, out=3, weight=-3.0010419817357072, enabled=True, innov=8) Node order: [4, 3]

  Output: expected 0.00000 got 0.11990
  expected 1.00000 got 1.00000
  expected 1.00000 got 0.92939
  expected 0.00000 got 0.00395 {2} {2} {0,
  1, 3, 4} {0, 1, 2, 3, 4}

However, when trying to implement the neural network in Python, I don't get the same results:

import math

def sigmoid(x):
    f = 1 / (1 + math.exp(-x))
    return f

node0 = 1
node1 = 1
node2 = 0

node3_bias = -2.340619398413843
node4_bias = 0.4674809078722945

connection_0_2 = 1.0534262789427578
connection_1_2 = -1.6479324848766828
connection_0_3 = -3.5479647367966765
connection_3_2 = 5.970619032226201
connection_1_3 = 4.634338591600734
connection_0_4 = 0.19203601809123594
connection_4_2 = -0.6588709650159562
connection_4_3 = -3.0010419817357072

def main():
    activated_node4 = sigmoid(node4_bias * 1 +
                              node0 * connection_0_4)
    activated_node3 = sigmoid(node3_bias * 1 +
                              activated_node4 * connection_4_3 +
                              node0 * connection_0_3 +
                              node1 * connection_1_3)
    activated_node2 = sigmoid(activated_node4 * connection_4_2 +
                              activated_node3 * connection_3_2 +
                              node1 * connection_1_2 +
                              node0 * connection_0_2)

    print("The value is: ", activated_node2)

if __name__ == "__main__":

    main()

For nodes 1 and 1 I get:

The value is:  0.30957048732254944

which does not match the expected output above.

Can anyone please help me with some pointers as to what I am getting wrong?

Thank you.

In case anyone has the same issue, please follow the github thread at:

https://github.com/CodeReclaimers/neat-python/issues/79


How much of a problem is each member of a batch having the same label? I have a batch size of 128 and a total data size of around 10 million, and I am classifying between 4 different label values.
How much of a problem is it if each batch only contains data with one label?
So for example - batch 0 all have the 3rd label.  Batch 1 all have the 1st.  Batch 2 the 2nd.  Etc.

If you are interested in convergence, then most probably it shouldn't converge. I used "should" because although we have an intuition of how the convergence work but no one is 200% sure of everything.
 Let's understand a few of the aspects of the whole learning process.
 - Every data point has a distinct Loss space
 - Overall Loss space will be the average of all individual Loss surface
 - We can safely assume, that the overall Loss space of the data points in one class will be closer to each other but very different when compared to the other class.
 - Lastly, the Gradient is calculated at the end of a Batch
When we have well-shuffled batch -
In this case, the Gradient after each batch will be directed towards the overall Gradient because the Gradient will be averaged. It means a smooth Loss reduction with each batch
When we have each batch of one Class -
 In this case, every alternate batch will move on two different Loss space(One of each Class).
The issue will be that we will calculate Loss for one space, but after the weight update at the end of a batch, the next batch will continue in its own space but using the weight from the previous batch. This will make learning very unpredictable.
 Also, many logic applied in optimizers will not help because Gradient will randomly change with each batch.
I tried this on MNIST digit (just using 0,1 digit) and got this Loss per Batch. I have not out enough effort on LR optimization, LR decay, etc. So, can't conclude that result will be always similar.
$\hspace{3cm}$

Welcome to StackExchange!
Yes, the idea of mini-batches is to augment balance in an unbalanced dataset.
You should train on balanced datasets (i.e same prevalence of all classes) and measure performance on a representative dataset as discussed here and here.

A neural network trained with imbalanced data often has varied levels
of precision in determining each class depending on the difference in
the number of class samples in the training data, which is a
significant problem in industrial quality inspection. As a solution to
this problem, we propose a balanced mini-batch training method that
can virtually balance the class ratio of training samples. In an
experiment, the neural network trained with the proposed method
achieved higher classification ability than that trained with
over-sampled or undersampled data for two types of imbalanced image
datasets.
[Ryota et al]


Train neural network for regression with negative samples I have training samples which have have vector $\vec x$ as input and a vector $\vec y$ as output - both vectors have real (float) numbers $\in \mathbb R$ as entries. I want to train a neural network such that if I put an $\vec x$ I get the right $\vec y$ out. Obviously I am having an input layer of as many neurons as $\vec x$ is large in dimension and output neurons as large as $\vec y$ is in dimension.

In my training set I have a lot of sample where I have a given input and and given output and I know that the output is not what I want (a wrong sample, or a negative sample if you will). Of course I could just not use those samples for training, but then I would through away some perfectly good information.

I know there are papers out there using negative samples for classification (categorical output), but I do not know of an approach for regression (continuous output).

How do I use my negative samples?

An option might be to tinker with the cost function: Usually one uses the difference between the target output ($out_{t}$) and the actual output ($out_{a}$) as a part as the cost function. Let's say

$$
J = \sum_{i=1}^N (out_{t,i} - out_{a,i})^2
$$

is the cost function, where N is the number of samples. Now we can adjust this to the cost function

$$
J = \sum_{i=1}^N type_i \cdot (out_{t,i} - out_{a,i})^2
$$

where $type_i$ is 1 for normal samples and -1 for "negative samples". Alternative one might also use

$$
J = \left( \sum_{i=1}^M (out_{t,i} - out_{a,i})^2 \right) + \left( \sum_{i=M+1}^N \frac1{(out_{t,i} - out_{a,i})^2} \right)
$$

as the cost function where the samples up to $M$ are the normal ones and from $M+1$ to $N$ are the negative samples.


What does it mean when we say most of the points in a hypercube are at the boundary? If I have a 50 dimensional hypercube. And I define it's boundary by $0<x_j<0.05$ or $0.95<x_j<1$ where $x_j$ is dimension of the hypercube. Then calculating the proportion of points on the boundary of the hypercube will be $0.995$. What does it mean? Does it mean that rest of the space is empty? If $99\%$ of the points are  at the boundary then the points inside the cube must not be uniformly distributed?

Speaking of '$99\%$ of the points in a hypercube' is a bit misleading since a hypercube contains infinitely many points. Let's talk about volume instead.

The volume of a hypercube is the product of its side lengths.
For the 50-dimensional unit hypercube we get $$\text{Total volume} = \underbrace{1 \times 1 \times \dots \times 1}_{50 \text{ times}} = 1^{50} = 1.$$

Now let us exclude the boundaries of the hypercube and look at the 'interior' (I put this in quotation marks because the mathematical term interior has a very different meaning). We only keep the points $x = (x_1, x_2, \dots, x_{50})$ that satisfy
$$
0.05 < x_1 < 0.95 \,\text{ and }\, 0.05 < x_2 < 0.95 \,\text{ and }\, \dots
\,\text{ and }\, 0.05 < x_{50} < 0.95.
$$
What is the volume of this 'interior'? Well, the 'interior' is again a hypercube, and the length of each side is $0.9$ ($=0.95 - 0.05$ ... it helps to imagine this in two and three dimensions).
So the volume is $$\text{Interior volume} = \underbrace{0.9 \times 0.9 \times \dots \times 0.9}_{50 \text{ times}} = 0.9^{50} \approx 0.005.$$
Conclude that the volume of the 'boundary' (defined as the unit hypercube without the 'interior') is $1 - 0.9^{50} \approx 0.995.$

This shows that $99.5\%$ of the volume of a 50-dimensional hypercube is concentrated on its 'boundary'.

Follow-up: ignatius raised an interesting question on how this is connected to probability. Here is an example.

Say you came up with a (machine learning) model that predicts housing prices based on 50 input parameters. All 50 input parameters are independent and uniformly distributed between $0$ and $1$.

Let us say that your model works very well if none of the input parameters is extreme: As long as every input parameter stays between $0.05$ and $0.95$, your model predicts the housing price almost perfectly.
But if one or more input parameters are extreme (smaller than $0.05$ or larger than $0.95$), the predictions of your model are absolutely terrible.

Any given input parameter is extreme with a probability of only $10\%$. So clearly this is a good model, right?
No! The probability that at least one of the $50$ parameters is extreme is $1 - 0.9^{50} \approx 0.995.$
So in $99.5\%$ of the cases, your model's prediction is terrible.

Rule of thumb: In high dimensions, extreme observations are the rule and not the exception.

The best way to "understand" it (though it is IMHO impossible for a human) is to compare the volumes of a n-dimensional ball and a n-dimensional cube. With the growth of n (dimensionality) all the volume of the ball "leaks out" and concentrates in the corners of the cube. This is a useful general principle to remember in the coding theory and its applications.

The best textbook explanation of it is in the Richard W. Hamming's book "Coding and Information Theory" (3.6 Geometric Approach, p 44).

The short article in Wikipedia will give you a brief summary of the same if you keep in mind that the volume of a n-dimensional unit cube is always 1^n.

I hope it will help.


How to find similar meaning log messages in the source code? Currently we have 10 thoundands log messages in the source codes which is maintained by different groups.
Some messages may be similar, but the sentence is not the same. My purpose is to create a centralized file which contains the unified message for reference.
For example:

printf("Failed to open the file %s", file)
printf("%s: failed to open file %s", TIMESTAMP, file)
printf("Failed to create the file %s")

I can extract all the message body which is in bold.They are not strict English sentence. but manually categorizing them takes too much time.
Is there a way to automatically categorizing the logs?
I just want to reduce the manual work.

A simple idea, non-supervision:
 1.Stemming, reduceing punctuations. Then you can use Bag of word with TF-IDF to represent the log sentences.

Run K-means to cluster your sentences. If your sentences are belong to
5 type. You can set K=5.

Of course, you can use Regular Express to classify those sentences.


Is over-fitting a matter of features engineering or a matter of parameters set? I am using sklearn package to make models.

I tried randomly to set some paramater to a sklearn.ensemble.RandomForestClassifier in order to have ideas about parameters to search for a future grid search.

Well, here the results of this attempt:

Accuracy :
Training: 0.9209427371993345 , Test: 0.7035561005518087

Clearly, I know it means the model is over-fitting because it's not able to generalize to new data. I know a cross-validation would be more accurate because it is possible the test-set is unfortunately too different from training-set just by random, but that's not what I am looking for in the current problematic of my topic.
Besides, the two classes I try to target are imbalanced. Class 1 is more present than Class 0.

Class 0: 34% of test set/training set, Class 1: 66% of test set/training set.

So because it's imbalanced I checked about precision and recall metrics:

On test set:

+---------+-----------+--------+----------+----------+
| classes | precision | recall | F1-score | support  |
+---------+-----------+--------+----------+----------+
|       0 |      0.60 |   0.38 |     0.47 |     3326 |
|       1 |      0.73 |   0.87 |     0.80 |    6460  |
+---------+-----------+--------+----------+----------+

On training set:

+---------+-----------+--------+----------+----------+
| classes | precision | recall | F1-score | support  |
+---------+-----------+--------+----------+----------+
|       0 |      0.98 |   0.81 |     0.89 |   31265  |
|       1 |      0.91 |   0.99 |     0.95 |  59492   |
+---------+-----------+--------+----------+----------+

Then, on training set I see both classes are well predicted. Intuitively I think: if it's well predicted on the training set, it means features are good enough to split the two classes. So, it's just a matter of parameters set.
But intuition is not as valuable as good experience is. So I am asking for more experienced people from this community if my intuition is wrong, and if so why it is?

UPDATES:

After checking variables, it shows distribution are the same between the training-set and testing-set through histograms.

A 1/3 - 2/3 repartition is not that unbalanced. Your problem shouldn't require balancing.

The train/test set partition seems to be done correctly, as it seems implied by checking data histograms. Doing that randomly is usually ok, and when it's not it will inflate your test performance with data leakage, which doesn't seems to be the case here.

Imo the problem come from your learning process, or more exactly : when does it stops ?

If you are going to explore your parameters space by hand, I would suggest to go look at the parameters of your learner : https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html, and for each one of them ask yourself what would be the impact of increasing / decreasing them on the learning capacity of your algorithm.

It may be a bit counterintuitive but you have to actcually limit the capacity of your learner, so that it does not learn too much.

Then you can do some grid search on the main parameters driving the learning / performance.

Features maybe good enough but obviously you have covariate shift, or some similiar distrubancd. In other words distribution of your train and test features is different and that confuses your model, in other words it doesnt learn to differentiate on train dataset.


Missing Values In New Data (Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified)

Suppose I've got a dataset which has 5 predictors - v1 - v5.
Since my dataset contains NA's, I've been imputing these values during training, which is fine.

However, if I get a new item to classify, it may contain 1 or more NA's. My initial thought was to create a model for each predictor variable; so for example, create a model where v2 - v5 predict v1, in the case that v1 is missing.

This would result in 5 additional models. One problem is that if v1 and v2 are missing, I'd need to guess both values somehow. I'm not sure how to do this. It makes me think that this isn't the best approach.

I'm a bit stuck, and any insight into this would be much appreciated :)

No expert here, but predicting missing value using existing variables seems suspicious to me. Does it really work?

Say $v_5$ is missing and you want to impute its value based the other variables available, using some arbitrary function $g$:
$$v_5 = g(v_1, v_2, v_3, v_4)$$
and the target variable T is:
$$T = f(v_1, v_2, v_3, v_4, v_5)$$
which is essentially
$$T = f(v_1, v_2, v_3, v_4, g(v_1, v_2, v_3, v_4))$$
or
$$T = h(v_1, v_2, v_3, v_4)$$
So essentially you are estimating $T$ using available information regardless of the missing value. Why bother to impute $v_5$ using arbitrary function?

I tend to think that imputation only makes sense only if you have a good reason for the value imputed -- often they come from domain knowledge, which means that you need to impose an external restriction on the functional and parametric form of $g$ based on your prior.


is it a good idea to take the derivative or integral of some features and add them as new features in machine learning? I'm learning how to do feature Engineering and come across some ideas in my head that's why I want to ask if I had some dataset with some features let's say 2 features and I have a timestamp column and the dataset is a time series dataset so they are monotonic. would it make sense to calculate the derivative or integral and add it as new feature ?

as an example let's say I have speed and acceleration as features, would it make sense to add the jerk (which is the derivative of the acceleration) and the snap(which is the derivative of the jerk) as new features ? also maybe the integral of the speed which would give the displacement I think?

the goal is let's say 2 features are not enough and we want to produce more features, is it wise to add the derivative or integral as a new feature? or is it a bad idea?

I also want to know whether the correlation between the derivative and integral according to timestamp and the feature that I derivated from would be high if I do this and is it bad or good if I make new features that correlate with others in my dataset

While performing feature engineering , transforming the existing features is recommended step instead of adding transformed values of the existing feature into the dataset.
Since, if we add these kind of data elements they will be highly correlated with each other and it effects the model performance.
If you have less features and cannot create a model, ask for additional data.


Machine Learning Validation Set I have read that validation set is used for Hyper-parameter tuning and comparing models. But, what if my algorithm/model does not have any hyperparameter? Should I use validation set at all? Because comparing models can be done using Test set also.

The validation set is there to stop you from using the test set until you are done tuning your model. When you are done tuning, you would like to have a realistic view of how the model will perform on unseen data, which is where the test set comes into play.

But tuning the model is not only hyperparameters. It involves things like feature selection, feature engineering and aslo the choice of algorithm. Even though it seems like you are already decided on a model, you should consider alternatives as it might mot be the optimal choice.

Comparing models cannot (or should not) be done using a test set alone. You should always have a final set of data held out to estimate your generalization error. Let’s say you compare 100 different algorithms.  One will eventually perform well on the test set just due to the nature of that particular data.  You need the final holdout set to get a less biased estimate.

Comparing models can be looked at the same way as tuning hyperparameters.  Think of it this way, when you are tuning hyperparameters, you are comparing models. In terms of requirements comparing random forest with 200 tress vs random forest with 500 trees is no different then comparing random forest to a neural net.


GridSearchCV Acting Weird I am using GridSearchCV to find the best combination of parameters for SVM. However, the parameters chosen by GridSeasrchCV do not seem to be the best ones. I tried some parameters randomly and they are working much better. I tried both combinations in the test set. Here is the picture:
The second confusion matrix + classification report is the random but improved one. Why isn't GridSearchCV selecting this combination?

99% Accuracy is over fitted model and its not the accurate model. There will always be loss associated. secondly you sample size is too small for reaching the global minima. you should also check your ROC-AUC curve for the model.


XGBoost outputs tend towards the extremes I am currently using XGBoost for risk prediction, it seems to be doing a good job in the binary classification department but the probability outputs are way off, i.e., changing the value of a feature in an observation by a very small amount can make the probability output jump from 0.5 to 0.99.

I barely see outputs in the 0.6-0.8 range. In all cases, the probability is less than 0.99 or 1.

I am aware of post training calibration methods such as Platt Scaling and Logistic Correction, but I was wondering if there is anything I can tweak in the XGBoost training process.

I call XGBoost from different languages using FFI, so it would be nice if I can fix this issue without introducing other calibration libraries, e.g., changing eval metric from AUC to log loss.

The first I'd ask would be "What is positive/negative ratios?". Just because I had a similar issue multiple times when classes were very imbalanced. If it's your case you can try to balance dataset or try to play with scale_pos_weight parameter of XGboost. In the case of 50/50 classes ratios, probablities might be more or less normalised.

Also, it's very likely you have overfitting at the same time. Try to tune learning rate, sampling parameters and regularization parameters (reg_alpha, reg_lambda).

First, you should be sure on that your data is large enough when working with tree-based algorithms like XGBoost and LightGBM, such sudden changes may indicate overfitting. (10,000 samples at least, rule of thumb)

Second, how is your cardinality; if you have 3-4 features, it would be expected that a change of feature causing such an affect.

Third, what are your selection of hyperparameters? Tree-based models are much sensitive to changes of the parameters. Be sure that you carefully implement your hyperparameter tuning.

Lastly, when dealing with binary classification; error metrics gets really important. You can do a combination of binary log loss and binary error (XGBoost allows you to choose multiple); also be sure to implement early stopping by choosing early_stopping_rounds = N in the train method of XGBoost, where N is the selection of iterations. By that, your algorithm will stop early at a reasonable point where your loss stops to decrease, avoiding overfitting.


How to predict an approximate weekly/monthly number, when the Unique Daily Visitors for that week/month are already known I am trying to come up with a formula or machine learning algorithm using which I can approximately predict the weekly or monthly users.

What to keep in mind is that I already have counts for the unique visitors per day for the week/months that I would like to make a near accurate prediction. Here, simply summing the daily unique users would not work, as they can be unique on one day but not on two days as they can have a session lasting over 2 days.

This method is to serve as an alternative to running a Spark job on the whole week/month data in order to save time and resources - Is this possible?

I have looked at Time Series and Linear Regression, but need more clarification on the possible approaches and also on any work-arounds?

To calculate the (estimated) value of the monthly/weekly unique visitors based on the per day count you may use the HyperLogLog algorihm.

This will give you the historical values and you can use any time series approach to predict future counts

I do not know what industry you are working in but predicting unique users can be difficult at times because of factors outside of your control. For example if your company tries a new advertising campaign that targets a new segment it is hard to calculate a range because they are a new eco system interacting with your previously defined set of rules.


How can I know if my conv1D model is overfitted or underfitted from loss curve? I am working on classification of time series multivariate data. By doing PCA, I converted multivariate to uni-variate and fed it into a conv1d in keras.

However, I am getting a very high accuracy and low loss both in validation and in training. How can I justify this?

I have tried cross validation, but the results are not much different.
I am using adam optimizer (learning rate:0.0001). With 0.001, my model fails to converge.

I have made sure that I am not mixing the training and validation datasets.
I have shuffle both datasets independent of each other. I trained on 3728 samples and validated on 610 samples.

Can we expect such a high accuracy with binary classification?

From the curves you are showing yes. Over-fitting would mean that your validation accuracy would be lower than your training accuracy, which is not the case here. Since you say that your training and validation sets are completely independent (i.e. no training samples are present in the validation set) you can consider the results reliable.

However accuracy might not be the best indicator of the model's performance. Make sure your dataset is balanced (i.e. the number of samples in both classes are equal to one another). If not try another metric that better represents the performance of your model.

Over-fitting happens when validation accuracy begins to decrease while training accuracy continue to increase (and the opposite for the loss).
Thus looking at your curves, there is no over-fitting because validation accuracy never decreases significantly.
If training and validation losses have very similar values, it just means that your training and validation data are very similar i.e. are drawn from the exact same distribution, which is a good thing.
It is possible to have very high accuracy in a binary classification problem if the 2 classes are very different and so easily separable.
For instance you might get very high accuracy in recognizing cat versus dog but not so high accuracy in recognizing cat breed 1 versus cat breed 2 if the 2 breeds are visually similar.


Clustering geo location coordinates (lat,long pairs) What is the right approach and clustering algorithm for geolocation clustering?

I'm using the following code to cluster geolocation coordinates:

import numpy as np
import matplotlib.pyplot as plt
from scipy.cluster.vq import kmeans2, whiten

coordinates= np.array([
           [lat, long],
           [lat, long],
            ...
           [lat, long]
           ])
x, y = kmeans2(whiten(coordinates), 3, iter = 20)
plt.scatter(coordinates[:,0], coordinates[:,1], c=y);
plt.show()

Is it right to use K-means for geolocation clustering, as it uses Euclidean distance, and not Haversine formula as a distance function?

K-means is not the most appropriate algorithm here.
The reason is that k-means is designed to minimize variance. This is, of course, appearling from a statistical and signal procssing point of view, but your data is not "linear".
Since your data is in latitude, longitude format, you should use an algorithm that can handle arbitrary distance functions, in particular geodetic distance functions. Hierarchical clustering, PAM, CLARA, and DBSCAN are popular examples of this.
This recommends OPTICS clustering.
The problems of k-means are easy to see when you consider points close to the +-180 degrees wrap-around. Even if you hacked k-means to use Haversine distance, in the update step when it recomputes the mean the result will be badly screwed. Worst case is, k-means will never converge!

Go with Kmeans clustering as HBScan will take forever.   I tried it for one of the project and ended but using Kmeans with desired results.


What's the difference between fit and fit_transform in scikit-learn models? I'm a newbie to data science, and I do not understand the difference between the fit and fit_transform methods in scikit-learn. Can anybody explain simply why we might need to transform data?
What does it mean, fitting a model on training data and transforming to test data? Does it mean, for example, converting categorical variables into numbers in training and transforming the new feature set onto test data?

To center the data (make it have zero mean and unit standard error), you subtract the mean and then divide the result by the standard deviation:

$$x' = \frac{x-\mu}{\sigma}$$

You do that on the training set of data. But then you have to apply the same transformation to your testing set (e.g. in cross-validation), or to newly obtained examples before forecast. But you have to use the exact same two parameters $\mu$ and $\sigma$ (values) that you used for centering the training set.

Hence, every sklearn's transform's fit() just calculates the parameters (e.g. $\mu$ and $\sigma$ in case of StandardScaler) and saves them as an internal object's state. Afterwards, you can call its transform() method to apply the transformation to any particular set of examples.

fit_transform() joins these two steps and is used for the initial fitting of parameters on the training set $x$, while also returning the transformed $x'$. Internally, the transformer object just calls first fit() and then transform() on the same data.

You don't want your model to learn anything from the test dataset. You just want to apply the learnings from your trained dataset. So, we only apply transform operation on test dataset and fit_transform operation on the train dataset.


Logistics regression with polynomial features vs neural networks for classification I am taking Andrew Ng's Coursera class on machine learning. He mentions that training a logistic regression model with polynomial features would be very expensive for certain tasks compared to training a neural network. Why is that though? I mean, when we talk about neural networks we're usually looking at a model with a very large number of parameters so why would logistic regression be more computationally expensive?

PS: Here's some context (at the beginning of an exercise on neural networks):

  In the previous part of this exercise, you implemented multi-class logistic regression to recognize handwritten digits. However, logistic regression cannot form more complex hypotheses as it is only a linear classifier (You could add more features, such as polynomial features, to logistic regression, but that can be very expensive to train).

I expect what he's referring to is the combinatorial explosion in the number of terms (features)
as the degree of the polynomial increases.

Let's say you have $N$ measurements/variables you're using to predict some other variable.  A $k$th degree polynomial of those $N$ variables has $ N+k \choose k$ terms (see here).  This increases very quickly with $k$.

Example:

Say we have N = 100 variables and we choose a third degree polynomial, we'll have
 $ 103\choose3$ = 176,851 features. For a five-degree polyomial it goes to:
$105 \choose 5$ = ~96 million features.  You'll then need to learn as many parameters as you have features.

Compare to a NN:

Compare this to using a fully connected NN
say we choose K fully connected hidden layers with $M$ units each.  That gives:
$ (NM) + (K-1)(MM) + M$ parameters.  This is linear in $K$ (though the $MM$ term attached to it might be big).

For N = 100 variables again, two hidden layers, and 350 features nodes per layer we get 157,580 parameters - less than we'd need for logistic regression with third degree polynomial features.

Representational power

(Added this section after seanv507's great comment)  * See caveat below

The argument above was just a numbers game - how big of a NN can you get while still having the same parameter count as logistic regression with polynomail features.  But you're getting at something when you say in your question:

  I mean, when we talk about neural networks we're usually looking at a
  model with a very large number of parameters so why would logistic
  regression be more computationally expensive?

Well said.

Efficiency

Neural nets are universal function approximators, and we know that polynomials can approximate functions too.  Which is better? What should one use? I'd bet that, for a given parameter "budget", a NN could better approximate "more" functions than a polynomial with the same number of parameters.  It wouldn't surprise me if there was theory to back it up, but I don't know it off hand.  seanv507's statement

  One possibility is to assume that the nonlinear activation function is quadratic...and identify what range of polynomials the NN could represent.

is an interesting idea.  Empirically, NN's have done better for many hard tasks than polynomial representations, and that's pretty strong evidence.

*Caveat

As seanv507 says - this is the hard part.  The above statements won't always be true - I'd argue they're probably mostly true.  If a low-degree polynomial basis nicely and reliably separates your classes, then it's probably worth using / trying polynomial features.

To my understanding, the meaning of Andrew Ng's words is: since logistic regressions are linear classifiers, the only thing you can do to improve their performance is to increase the number of variables (i.e. adding more fetaures). Which means that, in order to get the same results that you would get with a Neural Network, you'd need much more input data.

I 100% agree with you that the average Neural Network has way more trainable parameters than a logistic regression, but to reach the same performance of a Neural Network, a logistic regression would require much more of them.


Different number of features in train vs test I'm doing the titanic exercise on kaggle and there is a categorical Cabin attribute that has a lot of different strings: C41, C11, B20 etc. (about 100).

To be able to train my model I'm converting it to numerical attributes (using pandas get_dummies()). So in the end I get 100+ attributes.

On the test dataset however, there are less cabins, so I'll end up with fewer attributes.

I did something like this to make them equal (create columns that are in the training set and delete those that aren't):

for column in X.columns:
    if column not in X_test.columns:
        X_test[column] = 0

for column in X_test.columns:
    if column not in X.columns:
        X_test.drop([column], axis=1, inplace=True)

but I know it is not a good thing. So how else should I approach it?

I tried removing the cabin column altogether but my model performs better on test data with that column.

You could concatenate your train and test datasets, crete dummy variables and then separate them dataset.

Something like this:

train_objs_num = len(train)
dataset = pd.concat(objs=[train, test], axis=0)
dataset = pd.get_dummies(dataset)
train = copy.copy(dataset[:train_objs_num])
test = copy.copy(dataset[train_objs_num:])

You would prefer to concatenate data first and then convert in dummies followed by again splitting them in Training and Testing dataset as suggested by @Jekaterina Kokatjuhha.

Though the option of the merging of the dataset could be problem-specific, you may also read the article here for getting more understanding about having different observations within Train and Test Dataset.


What is the relationship between clustering and association rule mining? Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters).

Source: Wikipedia

  Association rule learning is a method for discovering interesting relations between variables in large databases.

Source: Wikipedia

So both, clustering and association rule mining (ARM), are in the field of unsupervised machine learning. Clustering is about the data points, ARM is about finding relationships between the attributes of those datapoints.

However, I wonder if there are more relationships. For example, given a clustering, can this enhance / simplify ARM or vice versa?

First of all, the problem of finding frequent itemsets (ARM) differs from the similarity search (clustering).

If you think about market baskets, clustering tries to find objects that have a large fraction of their baskets in common, the absolute number of those objects (basket occurences) is not of interest.
With ARM we are interested in the absolute number of baskets that contain a particular set of items.

CLustering:  Allocates objects in such a way that objects in the same group (called a cluster) are more similar (given a distance metric) to each other than to those in other groups (clusters).

ARM: Given many baskets (could be actual supermarket baskets) find which items inside a basket predict another item in the basket.

Sources

The paper

  Marie Plasse et al: Combined use of association rules mining and clustering methods to find relevant links between binary rare attributes in a large data set. Link

combines both, clustering and association rule mining. They could improve ARM by association rule mining. From the abstract:

  A method to analyse links between binary attributes in a large sparse data set is proposed. Initially the variables are clustered to
  obtain homogeneous clusters of attributes. Association rules are then mined in each cluster. A graphical comparison of some rule
  relevancy indexes is presented. It is used to extract best rules depending on the application concerned. The proposed methodology
  is illustrated by an industrial application from the automotive industry with more than 80 000 vehicles each described by more than
  3000 rare attributes.