Understanding Classifier performance on text data I am working on a multi-label text classification problem(Total target labels 90). The data distribution has a long tail and class imbalance and around 1900k records. Currently, I am working on a small sample of around 100k records with similar target distribution.I am using the OAA strategy (One against all). I have tried many algorithms on data.

Currently, each label has atleast 5000 data rows. The class imbalance is high with around 80k records for the most common label and the most rare with just one data row which I have not considered in the modelling. This dataset contains text from academic journals. It has Title and Abstract  columns.

I am using HashingVectorizer(number of features 2**20, char analyzer) to generate features and TSVD to reduce the dimensionality(n_components=200).

LinearSVC(class_weight='balanced') # Got many warnings that it could not converge. I came to know that it may due to data not scaled properly. How can I scale text data??
LogisticRegression(solver='lbfgs') # Converged very quickly
RandomForestClassifier(n_estimators=40,class_weight="balanced") # Train time ~2hr

I noticed that LinearSVC has good recall(less false negatives) while Logistic and RF has good precision(less false positives) scores. Can anyone help me in identifying the reasons behind these scores and how can I improve them.

Currently, I am not using deep learning/transformer models due to limited computation resources.

As the data is imbalanced and skewed towards few classes, that's why RF and Logistic results are biased causing high FP values, therefore high precision and low recall.
SVC on the other hand might have tried to create hyperplanes to get most out of the other side of curve, thereby producing different results.

To improve the results, try to use non linear kernels and also try to balance the input data (by scaling etc) before feeding to classifiers.

Classifier Analysis

Try to leverage this python library for more  insights:
https://github.com/marcotcr/lime
reference: https://marcotcr.github.io/lime/tutorials/Lime%20-%20basic%20usage%2C%20two%20class%20case.html

Deep Learning Specifics

For deep learning models, you can use  Google Collab if possible.

If the model is deep learning based and is built on allennlp, the Gradient visualization can be performed.
Refer this link for Demo on Sentiment Analysis Problem,
https://demo.allennlp.org/sentiment-analysis/MTc0MzQwOA==
Tutorial is at  https://allennlp-course.apps.allenai.org/.

Multi Label Specifics

Refer here https://scikit-learn.org/stable/modules/multiclass.html.

Looks the probelm you are addressing is Multi Label Classification.

This will give you class wise reports, sklearn.metrics.classification_report. Accordingly, the narrow down poorly performing classes. And then tweak the model.

There is Micro/Macro metric for Multi Label problems.
Refer here Micro Average vs Macro average Performance in a Multiclass classification setting

Suggestion
My suggestion is better to explore how the metric is calculated.
Find the fact that what F1 is being referred here.

Analyse F1, precision and recall, over labels.
Then, it can be found out which labels are performing poorly.


How to make a dataframe with lists or vectors as its elements This is something I have been wondering for ages but I am never able to get an answer.

I am trying to understand how to make a dataframe in R, where each element of the dataframe is itself a vector or a matrix.

For example, lets say we have a regular vector V with elements being real numbers.

Then to acess any number we would have

V[3] which would give the third element of said vector.

Now I want to know how to do this with say a dataframe D,

where each element of the dataframe is itself a vector or matrix, so that say

D[3] is not a real number, but a vector.

How can this be done in R?

Thanks all

Dataframe is a representation of 2 D matrix in R. R does not support MultiIndex dataframes that can represent more complex structures.

Maybe you can model the problem as a Tensor ( https://cran.r-project.org/web/packages/tensorr/vignettes/introduction.html , https://www.rdocumentation.org/packages/rTensor/versions/1.4/topics/Tensor-class)


Predicting missing data. Looking for good data predicting technique I am analysing data for Countries Trade GDP. Some of the countries have missing GDP value for given a year. However, I have Grand Total for the entire region for that year. Is there a good data science technique to fill the missing Trade in GDP for the below example.

there are several ways how to interpolate the missing data depending on how long history record do you have available:

you can compute ration of GDPs r_i (where all r_i sums to 1) in year Y-1 of each state with missing values in year Y; then you know the amount of GDP in year Y which should be spread among missing (GDP_total - GDP_known) and multiply by r_i
a more sophisticated way would be computing the gradient from the timeline Y-1, Y-2, Y-3, ... and you predict a value for year Y for each missing state; then you would again compute ration r_i among missing and distribute remaining GDP accordingly. This regression can be done also backward if you know Y+1, Y+2, Y+3, ... or interpolation from Y-1 and Y+1.

depending on the tools you’re using. But in Python I came across this forum where something similar was asked
https://stackoverflow.com/questions/41532518/fill-in-a-blank-dataframe-column-with-all-0-values-using-python

I hope this helps.


Measuring the uncertainty of predictions Given a multiclass classification model, with n features, how can I measure the uncertainty of the model for that particular classification?

Let's say that for some class the model accuracy is amazing, but for another it's not. I would like to find a metric that will let me decide if, for a particular sample, I would like to "listen" to the model or not.

I thought about using prediction intervals, but am not sure how to calculate them.

The model you will decide how best to get an uncertainty. If you used Bayesian optimisation (that's a great package for it in Python), for example, you get a covariance matrix along with your expected values, and so inherently get an uncertainty measure. In this case, you can make predictions as to the underlying function of your data, and the (co-)variance will provide levels of uncertainty, as shown by the width of the green bands around the line below:

So the red points show where we have some sample data... notice that we have none e.g. at X = 4 and X = -1, which is why we have high undcertainty; the 95% confidence interval is very large.

If you use a standard deep neural network tpo perform classification, for example, there is no inherent measure of uncertainty. All you really have is your test accuracy, to let you know how well the model performs on hold-out data.  I cannot remember where it is explained, but I believe it is not actually feasible to interpret the class prediction values in terms of uncertainty.

For example, if you are predicting cat or dog for an image, and the two classes receive (normalised) logit values [0.51, 0.49] respectively, you cannot assume this means very low certainty.

I think you looking for

model.predict_prob()

in python lots of models have it. and with this function, you can calculate how strong model sure about its answer.


How to use TimeDistributed fo CNN+LSTM? I am trying to classify 6 classes time-frequency domain signal (STFT spectrogram) with a size of 3601x217 pixels. Assume that for each classes have 70 training samples, 20 validation samples, and 10 test samples. Each sample is a PNG image of size 3601x217x1 (grayscale), while in the future I am thinking to expand this into 3601x217x4 (3 color channels + alpha). But right now, I am focusing on grayscale first.

After trying pure LSTM, I found out that the model overfits quickly (training accuracy > 90%, but val_acc stucks at 20%). ow I'd like to try out Time Distributed CNN+LSTM.

Following is the code:

img_width, img_height = 3601,217
train_data_dir = 'sensor1/training'
validation_data_dir = 'sensor1/validation'

num_classes = 6
nb_train_samples = 70
nb_validation_samples = 20
epochs = 50
batch_size = 5

if K.image_data_format() == 'channels_first':
    input_shape = (3, img_width, img_height)
else:
    input_shape = (img_width, img_height, 3)

model = Sequential()
model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu', input_shape = input_shape)))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Conv2D(32, (3,3), padding='same', strides=(2,2), activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Conv2D(64, (3,3), padding='same', strides=(2,2), activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(units=128, return_sequences=False))
model.add(LSTM(units=64, return_sequences=False))
model.add(Dense(32))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

train_datagen = ImageDataGenerator(rescale = 1. / 255)
test_datagen = ImageDataGenerator(rescale = 1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir, target_size=(img_width, img_height),
    batch_size=batch_size, color_mode='grayscale')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir, target_size=(img_width, img_height),
    batch_size=batch_size, color_mode='grayscale')

model.fit_generator(
    train_generator, steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs, callbacks=[plot_losses],
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size)

The code gives following error:

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\training.py",
  line 496, in _make_train_function
      raise RuntimeError('You must compile your model before using it.')

  RuntimeError: You must compile your model before using it.

What's wrong with the code? I actually compiled the model before calling model.fit_generator:

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

You are specifying the input_shape inside the Conv2D layer, meaning:
model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu', input_shape = input_shape)))

Instead you should write:
model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu'), input_shape = input_shape))

or more clearly:
model.add(TimeDistributed(Conv2D(...), input_shape=input_shape))

Furthermore I should note that you are using a TimeDistributed with a Conv2D. This implies that your input_shape should be like this (timesteps, dim1_size, dim2_size, n_channels).

You are using: input_shape=(img_width, img_height, 3)

If you want to take the img_width as timesteps you should use TimeDistributed with Conv1D.

To summarize, always consider that a TimeDistibuted layer adds an extra dimension to the input_shape of its argument-layer.

Lastly, your first LSTM layer with return_sequences=False will raise an error. You must give it a True value.


IDE for Machine Learning workflow - AWS SageMaker Looks like we don't really use an IDE in any of Machine Learning workflow stages if we use AWS SageMaker.
Entire work is done in jupyter notebook. Is this correct?

Some ide you can use in your local machine :

Spyder
Jupyter Notebook
Jupyter Lab

Pycharm VSCode also has intregated support for notebook but i would suggest jupyter lab. It's really good
If you want to run on VMs then google colab and kaggle kernel is really good for free.


Is it possible for a DCGAN to do regression? What are some examples of this? I'm currently a student doing some machine learning projects, and I want to use generative adversarial networks to train some data to discern for example, how old someone is. The intended output is a regression continuous output in term of months.

I see the DCGANs and GANs are targeting mainly classification problems. Can they be used to solve regression problems?

Thank you.

The answer is yes.  There is a paper Intra-class Variation Isolation in Conditional GANs (https://arxiv.org/pdf/1811.11296.pdf) discussing it.

They propose a "C-GAN that is able to learn realistic models with continuous, semantically meaningful input parameters".  They actually cover generating images of people at different ages as well.

I am also interesting about it. Recently, I found a paper which the authors used a conditional GAN for regression tasks, in hope it will drive you over this problem. For a detail about this paper, see here.


How to model anomaly data using Gaussian distribution assuming variables are dependent? (In Python) I have some data which contains anomalies as well.

I want to model data using Gaussian distribution assuming variables are dependent in Python.

How can I model this? Should I use the PDF formula as a function, figure out the probabilities and take out the accuracy?

Note: This has been given as an assignment in my college. I am trying to figure out the solution.

If you assume variables are dependent, in a Gaussian distribution this means the covariance matrix is not diagonal.

You can start with considering the multi-dimensional Gaussian distribution. Its density is
$$ f(x_1,x_2,...,x_n) = \frac{1}{\sqrt{(2\pi)^n|\Sigma|}}\exp\left(-\frac{1}{2}(x - \mu)\Sigma^{-1}(x-\mu)\right)$$
(see https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Non-degenerate_case)
I suggest using the maximum likelihood approach to tune the parameters or any anomaly detection algorithm you know.


Extracting data from a data frame with conditions Given the first data frame consisting of 4 columns

libId    studId    year    freq
 1000      3       2002     3
 1000      34      2002     2
 1000      52      2004     9
 1001      17      2003     5
 1001      34      2003     1
.
.
.
.

If I were to extract only data with maximum frequency based on year for each libId this would be the desired output.

 libId    studId    year    freq
 1000      3       2002     3
 1000      52      2004     9
 1001      17      2003     5
.
.
.
.

So far I've tried

data2<- by(data1, c(data1[c(1,2,3)]), function(df) max(df$Freq))

data2 <- as.data.frame(as.table(data2))

But my output is incorrect:

libId    studId    year    freq
 1000      3       2002     3
 1000      34      2002     2
 1000      52      2004     9
 1001      17      2003     5
 1001      34      2003     1
.
.
.
.

There is a good summary on StackOverflow from the user EDi that I'm referring to. Basically, what you want to do is group the values by year and then select the maximum frequency of each group:

# with dplyr
require(dplyr)
df %>% group_by(year) %>% summarise(freq = max(freq))

# data.table
require(data.table)
dt <- data.table(df)
dt[ , max(freq), by = year]

Check out the linked answer for further possibilities.

You can try using subset function.
Below is an example.

output_df<- subset(input_df, freq>= 3)


XGBOOST missing_value feature degrades my performance? I'm training an xgboost model for gout disease on a training set I sampled 1-to-7 case-control ratio (enriched in cases). I have 220 features and I reach a cross-validated AUC of 0.90. I'm using a special value -65336 for missing values and I don't tell that to XGBOOST - I let it treat missing values just like any other value.

I then use it on the general population, with the true ratio of about 1-to-13 case-control. I get slightly worse AUC 0.84, and my prediction means is a reasonable 0.14.

THEN, I tell XGBOOST that missing_value=-65336:

the cross-validated performance on the train population stays the same.
BUT the performance on the general population degrades from AUC 0.84 to AUC 0.79. Also, I noticed that the prediction mean is changing from 0.14 to 0.40(!)

Why is this dramatic degradation happening? Should I refrain from using the missing-value feature when transferring between slightly different populations? has anyone else encountered this behavior?

If you have more than 50% of values as missing across any columns you should impute only when you are an SME of the domain or know the reason for missing(Honestly I would drop that column rather imputing). Not only XGboost most of the algorithm would penalise for Missing values and one can increase the performance by imputation. Tip may be try Catboost


Data mining- Clustering techniques I have a project for comparison between clustering techniques using the data set of SSA for birth names from 1910-2013 years for the different states.

I have finished applying my clustering techniques on my data set and the output of the clusters were the clusters of the states for each year.

Now I can know from my results; which states are close to each other with the birth names and which were not. By looking

I would like to have interesting results to make my project report interesting, Which states are similar in birth names are not enough to make the read be excited about my project.

My questions:

any ideas of what can be learned from my project?
How can I show the comparison between the clustering techniques? anyway other than seeing how Homogeneous the clusters are?

It's common to perform clustering on certain features, and then use characteristics such as mean differences between other features (not used in clustering) to show how clusters differ.  It's not always meaningful to say that clusters differ merely by lower distances between cluster centers and objects assigned to them (as in k-means).

As an example, you might acquire other features (variables) for the states and then average each feature among objects assigned to each cluster.  Then show bar charts of averages for those variables among all the clusters to assist in explaining how the clusters differ.  You might also test for significantly different means of the supplemental features across clusters.

To compare clustering results from different clustering techniques you can use the Entropy Scorer node in KNIME (www.knime.org)
THis node overlaps and compares results from different techniques. The quality measure at the end tells you how similar the clusterization is.

-- Rosaria


Experiment click to lead prediction with Azure ML I am experimenting now with the Azure ML Studio and I am trying to predict leads based on the clicks I have.

I am exporting a data set of 60.000 Clicks and 8.000 Leads from these clicks.

My data set:

With the "Edit Metadata" I have transformed all the features, except the hasLead in Categorical Feature.

I am testing as many combinations as I can with the data, but my "Recall" is very bad. Do you have some suggestions about what kind of missing data maybe can increase the Recall?

Here the evaluation of the model:

This looks like a two class boosted decision classifier.  That should perform quite well.  I used it several times about 2-3 months ago and got excellent results.  I'm guessing you are not training/testing the right features.  What is your target variable?  Is it 'hasLead'?  I'm not sure what will be statistically influential on that dependent variable, but you  can run a feature engineering exercise to see what independent variables have the most influence.  Run the code below; change to suit your specific needs (obviously, use your specific data).

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# matplotlib inline

df = pd.read_csv("https://rodeo-tutorials.s3.amazonaws.com/data/credit-data-trainingset.csv")
df.head()

from sklearn.ensemble import RandomForestClassifier

features = np.array(['revolving_utilization_of_unsecured_lines',
                     'age', 'number_of_time30-59_days_past_due_not_worse',
                     'debt_ratio', 'monthly_income','number_of_open_credit_lines_and_loans',
                     'number_of_times90_days_late', 'number_real_estate_loans_or_lines',
                     'number_of_time60-89_days_past_due_not_worse', 'number_of_dependents'])
clf = RandomForestClassifier()
clf.fit(df[features], df['serious_dlqin2yrs'])

# from the calculated importances, order them from most to least important
# and make a barplot so we can visualize what is/isn't important
importances = clf.feature_importances_
sorted_idx = np.argsort(importances)

padding = np.arange(len(features)) + 0.5
plt.barh(padding, importances[sorted_idx], align='center')
plt.yticks(padding, features[sorted_idx])
plt.xlabel("Relative Importance")
plt.title("Variable Importance")
plt.show()


Forecasting one time series with missing data with help of other time series I have time series $R$, which shows, how something changes at the regional level.

I have several time series $U_i$, which show, how something changes at a special unit $I$ level.

There are many units in the region. $R$ has no missing data. Different $U_i$ have their own missing periods.

I want to forecast $U$ after a missing period using information of $R$ and information of $U$  when it was availible.

My thoghts till now:

Suppose $R$ is known on interval $[0, 365]$. Suppose $U_i$ is known on interval $[0,300]$. Let's take $R$ and $U_i$ both on interval $[0,300]$, take difference between them and trying to predict that difference with linear regression. So for interval $[301,365]$, I will have differences and to restore $U_i$, I will just have to take out my differences from $R$.

I don't like my solution, because:

We need a model for each $U_i$.
Because, sometimes data is more sparse and I don't even have a $[0,300]$ known interval, so not able to train regression properly.

Not a fully complete answer, but some inputs.

Your time series are correlated.
I assume that the measure you want to forecast for a region is an aggregation of units forecasts.

To address the first point, I usually use Vector Autoregressive Model (VAR) that forecast all time-series at once (each one being expressed as a regression using the others)
The second point involves the concept of hierarchical forecasting and reconciliation. You can exploit the fact that the regional forecast should/must equal the unit- forecasts. There can be a process to adjust forecasts to take that into account.

There are both packages for VaR and hierarchical reconciliation in R but as far as I know no direct code to handle both at the same time...

You may find this paper providing some details on the proposed approach:
https://mpra.ub.uni-muenchen.de/76556/1/MPRA_paper_76556.pdf

Looks like casual Impact library could help you with that
http://google.github.io/CausalImpact/CausalImpact.html
Or a at least principles from it


Can an R^2, or coefficient of determination be used on non-linear data? I have used the $R^2$ metric to determine how well my neural network performs a non-linear regression. And it seems to work. The plots look almost identical, and I get an $R^2$ value of 0.93... it seems to work perfectly,  however I have been told that $R^2$ should only work for linear data. Is this true or false?

The $R^2$ calculation for nonlinear regression models can very a lot depending on the software / application / etc. Also, they involve a lot of approximations in order to make any calculation and are not as easily interpretable as the linear case. There are many other methods that will give you a good and very interpretable result directly from your data. For example, some kind of cross-validation like K-fold cross-validation or bootstrap methods give you a good sense of how trustworthy your estimates are directly from your data without relying on a bunch of normal approximations. Lastly, for most applications I would also recommend using some goodness of fit measure that takes into account parsimony like the AIC or BIC.

If it's a linear regression (linear in the parameters), then yes the $R^2$ is fine and is interpretable in the commonly thought of way.

$R^2$ is good for cases where you have no model for the noise in your measurement. A good way to think of it is just as a measure of the fraction of the variance in the measurement that's covered by your fit model. But it's not necessarily what you want for cases where you're concerned you might be overfitting your data (i.e. using a fit function with more parameters than you can justify) or where the error bars on different data points are very different from one another (as happens, for example, when Poisson noise dominates). Chi2, and especially reduced chi2 (i.e. chi2 per degree of freedom) are probably more appropriate for these cases.


Unsupervised encoding of categorical features I have multiple log records with discrete categorical features. Shape of my dataset is (100k, 24) My aim is to look for anomalies in these records. I am planning to cluster the data after encoding.

Before going forward with any analysis, the categorical features need to be encoded; I'd have ideally gone with LabelEncoder or OneHotEncoder but the issue is some features have > 40 possible values.

As of now, I've frequency encoded the variables but is there something I am missing and is there a better way of going about this?

High cardinality is a common issue that we run into when doing encoding for categorical variables. One hot encoding and label encoding are the most common approaches however label encoding maynot be ideal if the variable is not nominal (not inherently hierarchical) and one hot encoding will lead to curse of dimensionality when no of categories are high. REplacing with ratio or looking at spectral encoding is a good approach to circumvent this

Details of spectral encoding are :-
https://towardsdatascience.com/spectral-encoding-of-categorical-features-b4faebdf4a

you need to use OneHotEncoder() for you independent variables (X),  and LabelEncoder() for your dependent variable (y).

If you want to reduce the number of dummy variables (output of OneHotEncoder) you can remove one dummy variable for each category. For example, for a categorical variable, Gender, you will have Gender_Female and Gender_Male; but only one is required. Gender_Female=0 means Male. So you need to use drop='first' for OneHotEncoder(). However, when you use this feature, you have to use handle_unknown='error' (the latest version has a limitation than you can't use handle_unknown='ignore'). I highly recommend you to read this article Choosing the right Encoding method-Label vs OneHot Encoder.

The other possible way to reduce the number of variables, but I don't recommend', is using feature selection methods, SelectKBest(). However, I don't believe it works for non-linear problems as I haven't got good results in my problems. You can read here more 1.13. Feature selection

A better solution may be dimensionality reduction methods, e.g. PCA, to reduce the number of apparent variables and make your modelling computationally lighter and faster; but you know that you may lose some accuracy when you use this method. Read here 6.5. Unsupervised dimensionality reduction


Unable to open .json file in pandas I want to convert a json file into a dataframe in pandas (Python). I tried with read_json() but got the error:

UnicodeDecodeError:'charmap' codec can't decode byte 0x81 in position 21596351:character maps to <undefined>

I think I have some unwanted data in the json file like noise. The data is server generated.

This is a collection from the json file:

{"_id":{"$oid":"57a30ce368fd0809ec4d1b41"},"session":{"start_timestamp":{"$numberLong":"1470151881189"},"session_id":"8356bd90-20160802-153121189"},"metrics":{},"arrival_timestamp":{"$numberLong":"1470152028294"},"event_type":"OfferViewed","event_timestamp":{"$numberLong":"1470151943271"},"event_version":"3.0","application":{"package_name":"com.think.vito","title":"Vito","version_code":"5","app_id":"7ffa58dab3c646cea642e961ff8a8070","cognito_identity_pool_id":"us-east-1:4d9cf803-0487-44ec-be27-1e160d15df74","version_name":"2.0.0.0","sdk":{"version":"2.2.2","name":"aws-sdk-android"}},"client":{"cognito_id":"us-east-1:1d507b8f-857c-42a4-a705-8db07d46bc8f","client_id":"aa092911-b9a7-498a-82da-76318356bd90"},"device":{"locale":{"country":"US","code":"en_US","language":"en"},"platform":{"version":"5.1.1","name":"ANDROID"},"make":"Xiaomi","model":"Redmi Note 3"},"attributes":{"Category":"90000","CustomerID":"4077","OfferID":"11846"}}

You have to read the file line by line, you can find a detailed answer in this question of stackoverflow

You can try this.

import pandas as pd
#read line by line
with open('json_file','rb') as f:
  entries=f.readlines()
lines=list(entries)
Cleaned=[str(line).rstrip() for line in lines]
#Removes \n
Json="[" + ','.join(str(cl) for cl in Cleaned) + "]"

pd.read_json(Json)


Optimizing parameters for a closed (black-box) system I am working on a problem that involves finding optimal parameter values for a black-box system. This system consists of 6 inputs and produces a single output. I can compare the returned values to observed values to determine whether the system is well calibrated based on the parameter values I specified.

The main problem that I am having is that I do not know how these parameters are used in the black-box system to produce the output. The parameters may be correlated so changing the value of one parameter may affect the way the others behave in the system.

Since I do not know the function I am not sure how to optimize this problem efficiently.

Question: what are some methods for optimization when the function is unknown?

Edits:

The variable types are each a vector of real numbers but we can make some assumptions if it is is helpful.

The cost to run the black-box system is only time. Let's say that it takes 10 seconds to run the system but I happen to have five black boxes -- so if I can run my algorithm in parallel then I can cut down the run time. If the algorithm runs sequentially then I don't gain anything by having the extra boxes (expect perhaps choosing different starting positions).

For this problem I am interested in learning about how to solve this type of problem -- not trying to brute force my way to a solution. That is to say I am less interested in obtaining an answer and more finding out how I would obtain the the answer. I think it is easy to say that given sufficient resources (time, processing power) that simply generating many many random numbers and choosing the combination that gives the best result is the simplest (and probably worst) approach.

There are lots of function optimising routines that could be applied, based on the description so far. Random search, grid search, hill-climbing, gradient descent, genetic algorithms, simulated annealing, particle swarm optimisation are all possible contenders that I have heard of, and I am probably missing a few.

The trouble is, starting with next to zero knowledge of the black box, it is almost impossible to guess a good candidate from these search options. All of them have strengths and weaknesses. To start with, you seem to have no indication of scale - should you be trying input parameters in any particular ranges? So you might want to try very crude searches through a range of magnitudes (positive and negative values) to find the area worth searching. Such a grid search is expensive - if you have $k$ dimensions and want to search $n$ different magnitudes, then you need to call your black box $n^k$ times.

This can be done in parallel though, and given you are confident that the function is roughly unimodal, you can start with a relatively low number of n (maybe check -10, -1, 0, +1, +10 for 15625 calls to your function taking roughly 8 hours 40 mins using 5 boxes). You may need to repeat with other params once you know whether you have found a bounding box for the mode or need to try yet more values, so this process could take a while longer - potentially days if the optimal value for param 6 is more like 20,000. You could also refine more closely, once you have a potential mode you might want to define another grid of values to search based around. This basic grid search might be my first point of attack on a black box system where I had no clue about parameter meaning, but some confidence that the black box output had a rough unimodal form.

Given the speed of response you should be storing all input and output values in a database for faster lookup and better model building later. No point repeating a call taking 10 seconds when a cache could look it up in 1 millisecond.

Once you have some range of values you think that a mode might be in, then it is time to pick a suitable optimiser.

Given the information so far, I would be tempted to run either more grid search (with a separate linear scaling between values of each param) and/or a random search, constrained roughly to the boxes defined by the set of $2^6$ corner points around the best result found in initial order-of-magnitude search.

At that point you could also consider graphing the data, to see if there is any intuition about which other algorithms could perform well.

With the possibility of parallel calls, then gradient descent might be a reasonable guess, because you can get approximate gradients by adding a small offset to each param and dividing difference that causes in the output by it. In addition, gradient descent (or simple hill climbing) has some chance of optimising with less calls to evaluate the function than approaches that rely on many iterations (simulated annealing) or lots of work in parallel (particle swarm or genetic algorithms). Gradient descent optimisers as used in neural networks, with additions like Nesterov momentum or RMSProp, can cope with changes in function output "feature scale" such as different sizes and heights of peaks, ridges, saddle points.

However, gradient descent and hill climbing algorithms are not robust against all function shapes. A graph or several of what your explorations are seeing may help you to decide on a different approach. So keep all the data and graph it in case you can get clues.

Finally, don't rule out random brute-force search, and being able to just accept "best so far" under time constraints. With low knowledge of the internals of the black box, it is a reasonable strategy.

Here is a solution.

A mathematical method is described in this paper


K-modes clustering: Estimating which features were most impactful on clustering? I have entirely categorical data (survey results from users), so I've used k-modes clustering to better understand my users.

I'm not an expert at clustering methods at all. Is there a way to known way of estimating the importance of a feature (or combination of features) in deciding which cluster a user falls into?

It boils down to counting.

Compute which feature value has the highest probability of predicting a particular cluster. It's a straightforward application of Bayes' formula.

Read about PCA("Principal Component Analysis") and while implementing must standardize data on similar scale.


How to download COCO dataset images? I'm trying to download the COCO dataset images using the following COCO API command:

from pycocotools.coco import COCO
import requests
catIds = COCO.getCatIds(catNms=['person','dog', 'car'])

... but getting the following error message. Any idea why is it so?

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-42-9cc4e2f62a0e> in <module>
----> 1 catIds = COCO.getCatIds(catNms=['person','dog', 'car'])

TypeError: getCatIds() missing 1 required positional argument: 'self'

Previously I configured pycocotools using the following instruction:
https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/62381

COCO is a python class and getCatIds is not a Static Method, tho can only be called by an instance/object of the Class COCO and not from the class itself.

You can probably solve it by doing this instead:

a = COCO() # calling init
catIds = a.getCatIds(catNms=['person','dog', 'car']) # calling the method from the class

(answer based on reading the code from https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/coco.py)

Still some strange error messages. Looking into code it seems as if there are no such categories as a 'person','dog' or 'car'. Why is it so?

My code:

a = COCO()
catIds = a.getCatIds(catNms=['person','dog', 'car'])

Received error message:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-12-57400207fde1> in <module>
      1 a = COCO() # calling init
----> 2 catIds = a.getCatIds(catNms=['person','dog', 'car'])

~\Anaconda3\lib\site-packages\pycocotools\coco.py in getCatIds(self, catNms, supNms, catIds)
    171             cats = self.dataset['categories']
    172         else:
--> 173             cats = self.dataset['categories']
    174             cats = cats if len(catNms) == 0 else [cat for cat in cats if cat['name']          in catNms]
    175             cats = cats if len(supNms) == 0 else [cat for cat in cats if cat['supercategory'] in supNms]

KeyError: 'categories'

Extract from the file 'cocoapi/PythonAPI/pycocotools/coco.py':


Classifying Car Data By Year I have huge car photos.

I want to predict car's "brand-model-body type and production year"

First, I splitted data into train and validation, and I categorized them like this.

Every category has about 1000 train and 900 validation images.

My plan was: I train my keras model with these categories after training, model can predict labels like below:

audi a3 sedan 2008  => %25

audi a3 sedan 2009  => %25

audi a3 sedan 2010  => %25

audi a3 sedan 2011  => %25

And I can tell user that: "This car is Audi A3 Sedan 2008-2011"

My problem is, some of these categories have very similar photos. For example: audi a3 2009 and audi a3 2010 have same body type and there is not much difference between photos (No difference in reality).
Because of that, train accuracy has improved to about 0.9 but validation accuracy hasn't improved above 0.55

When I try some predictions, it usually gives same label, "Ford Focus sedan 2009" :)

Here is my output:

epoch,          acc,              loss,              val_acc,       val_loss
27,    0.7965514530544776, 0.56618134500483, 0.5192149643316993, 1.729015349846447

28,    0.8058803490480816, 0.5408204138258657, 0.5176764522193236, 1.778763979018732

29,    0.8167710489770164, 0.5116128672937693, 0.523258489762041, 1.7806432932022545

30,    0.8256544639818643, 0.4872381848016096, 0.5207534764479939, 1.8059904007678271

31,    0.8355546238309248, 0.4629556378035959, 0.5237253032663666, 1.8191414148756815

32,    0.8424464767701014, 0.4444190686917562, 0.5242512903147193, 1.8496954914466912

33,    0.8508739288802705, 0.422022156655134, 0.5303593149032422, 1.8565427863780883

34,    0.8576819265745635, 0.40545297008116027, 0.5262894901236571, 1.909881308499735

My train code is here:

Image_width, Image_height = 224, 224
num_epoch = 5000
batch_size = 16
learning_rate = 0.0001
model = ResNet50(weights='imagenet', include_top=False, input_shape=(Image_width, Image_height, 3))
fc_neuron_count = 1024
output = model.output
output = GlobalAveragePooling2D()(output)
output = Dense(fc_neuron_count, activation='relu')(output)
predictions = Dense(num_classes, activation='softmax')(output)
model = Model(inputs=model.input, outputs=predictions)

model.compile(optimizer=opt.Adam(lr=learning_rate), loss=losses.categorical_crossentropy,
              metrics=['accuracy'])

history_transfer_learning = model.fit_generator(
    train_generator,
    epochs=num_epoch,
    steps_per_epoch=num_train_samples // batch_size,
    validation_data=validation_generator,
    validation_steps=num_validate_samples // batch_size,
    class_weight='auto',
    callbacks=callbacks_list)

Am I doing something wrong? How can I achieve this result?
Should I change validation accuracy calculation, or should I give more photos per category?

I want to say how I solved my problem for anyone who is looking for a similar question.

My categorization was a mistake. I realized later that, I gave the same photos to my model and waited for the different results. For example, I had nearly same photos in Audi A3 Hatchback/5 2009 and Audi A3 Hatchback/5 2010. When the model starts training, first it learns data. Then it predicts and validates output itself. If the output is wrong, it tries a different way to success. But wait a minute, there was no mistake. I gave you the same photos and waited for different results! My categorization failed here.

I categorized my cars by body changes like Audi A3 Hatchback 2008-2013. Except for categories that have the wrong photos, my results are great for now.

Now, we will work on better photos and better year categorization.


What best/correct algorithm/procedure to cluster a dataset with a lot 0's? I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation.

I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) grouping them considering the number of events occurred.

My concern is that a lot of the values are 0 simple because some diseases didn't show that particularly symptom, for example (I made up the values for now):

So, I was wondering what was the best way to cluster this dataset. I was looking for and found hierarchical and kmeans, but don't know if i can properly apply to this cenario.
First of all, I switched the absolute values of occurrences to % of total, does that make possible to deal with the 0's ? I thought about it but at the same time 1% is close to 0%, I don't know if the algorithm can also understand as 'flags', since 1% represents that actually that symptom occurr(even at lower rates) and in another disease don't occurr at all.

I heard about PCA to reduce number of variables, and I was also curious if:
1-PCA is applicable to this cenario (dataset with a lot of sparse 0's)
2-PCA could solve my problem(because I think that even when a reduce to 2 3 variables, some rows could still be 0 for that particular column(symptom).

Any help/guidance would be extremely helpful and I thanks all in advance,
Sorry for some english error as well!

Have a great week!

Maybe I am not understanding your question properly, but k-means clustering is not sensitive to the zeros. More accurately zeros are valid values for features to cluster on.

While PCA may enable you to reduce the number of features to train on, it may complicate the clarity of the clusters from k-means. PCA may be beneficial in its own right, but AI would resist the urge to feed PCA data into k-means simply to reduce the number of zeros in your data.

HTH

If I understand your problem statement correct, You want to cluster data which have a lot of 0's (which mean your data is not balanced).
You can use any clustering technique (Supervised or Unsupervised) you like and plot the data to visualize.

If you want to train your model for future prediction,  I would suggest to balance the data before you start training (fitting) your chosen model.

You can use resample to scale-up of scale-down your data, once done you can concatenate and create final dataset for your model training.
from sklearn.utils import resample

Thanks!


Weight-initialisation and model instability Trying to calibrate a relatively vanilla NN, setting the hyper-parameter tuning aside*, it appears that weight initialisation has a lot of impact on the model output. Ie. Models calibrated with differents weight initialisations, despite similar overall performance, can yield very different individual prediction.

What could be the causes of such a behavior ? What would be the solutions ?

*The hyper-parameter tuning can't really be set aside, as regularisation may help getting the same result by reducing the number of variables. However, I observe similar behavior if I use a value significantly above / below the optimal hyper-parameter value.

As you have noticed weight initialization is extremely important, If for instance, you initialize your network with zeros, then it's guaranteed you will not learn from data. there are many ways to initialize your data:

Most simple approach (beginner approach) :
just use random initialization

Xavier Initialization:
this might be a more intermediate approach if you are interested on this check the following link:
https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79

I'd say that you gotta keep in mind when your network is "learning" it is basically navigating in a hyper dimension. most people do not actually standardize data. Standardizing data is essential since you make learning easier for your algorithm.

So short answer: standardize your data, your algorithm will converge faster into a minimum (of course not always global).


This was a x-post. There is a great solution over here, where I originally asked the question.

  If I understand your problem, you'll want something like this:

bottom <- seq(1750, 2010, 5)
library(dplyr)
new_df <- mlt.mx.info %>%
  arrange(Year) %>%
  mutate(year2 = as.numeric(substr(Year, 6, 9))) %>%
  mutate(new_year = paste0(bottom[findInterval(year2, bottom)], "-",(bottom[findInterval(year2, bottom) + 1] - 1)))
View(new_df)

  So what this does, it creates bins, and outputs a new column
  (new_year) that is the bottom of the bin. So everything from 1750-1754
  will correspond to a new value of 1750-1754 (in string form; the
  original is an integer type, not sure how to fix that). Does this do
  what you want? Double check the results, but it looks right to me.
  @goodtimeslim

Use library(reshape) for most of the pivot & filter operations. Should resolve your problem.


What are the use cases for Apache Spark vs Hadoop With Hadoop 2.0 and YARN Hadoop is supposedly no longer tied only map-reduce solutions. With that advancement, what are the use cases for Apache Spark vs Hadoop considering both sit atop of HDFS? I've read through the introduction documentation for Spark, but I'm curious if anyone has encountered a problem that was more efficient and easier to solve with Spark compared to Hadoop.

Hadoop means HDFS, YARN, MapReduce, and a lot of other things. Do you mean Spark vs MapReduce? Because Spark runs on/with Hadoop, which is rather the point.

The primary reason to use Spark is for speed, and this comes from the fact that its execution can keep data in memory between stages rather than always persist back to HDFS after a Map or Reduce. This advantage is very pronounced for iterative computations, which have tens of stages each of which is touching the same data. This is where things might be "100x" faster. For simple, one-pass ETL-like jobs for which MapReduce was designed, it's not in general faster.

Another reason to use Spark is its nicer high-level language compared to MapReduce. It provides a functional programming-like view that mimics Scala, which is far nicer than writing MapReduce code. (Although you have to either use Scala, or adopt the slightly-less-developed Java or Python APIs for Spark). Crunch and Cascading already provide a similar abstraction on top of MapReduce, but this is still an area where Spark is nice.

Finally Spark has as-yet-young but promising subprojects for ML, graph analysis, and streaming, which expose a similar, coherent API. With MapReduce, you would have to turn to several different other projects for this (Mahout, Giraph, Storm). It's nice to have it in one package, albeit not yet 'baked'.

Why would you not use Spark? paraphrasing myself:

Spark is primarily Scala, with ported Java APIs; MapReduce might be friendlier and more native for Java-based developers
There is more MapReduce expertise out there now than Spark
For the data-parallel, one-pass, ETL-like jobs MapReduce was designed for, MapReduce is lighter-weight compared to the Spark equivalent
Spark is fairly mature, and so is YARN now, but Spark-on-YARN is still pretty new. The two may not be optimally integrated yet. For example until recently I don't think Spark could ask YARN for allocations based on number of cores? That is: MapReduce might be easier to understand, manage and tune

Machine learning is a good example of a problem type where Spark-based solutions are light-years ahead of mapreduce-based solutions, despite the young age of spark-on-yarn.


Featuring scaling whole data set before spliting it. I am wondering why do we use scaling on train and test set separately. I understand that transform () on test data  μ and σ as computed from fit_transform() on  Train. But why can we compute μ and σ from all given data (before split) and then apply them on future data.

Do we do this because we don't know how the size of our future data?

You must use the same scaling factors (μ and σ) for training and test data. The reason that you must use the same scaling factors is that you should compare the data with the same pre-processing. Note that scaling factors (μ and σ) must be computed using only the training data. That is because in practice you are not provided with test data and you just have to evaluate your model on test data.

train and test datasets should have no overlap of data ... so when you scale each they may very well have different scaling factors


Improving LSTM Time-series Predictions I have been getting poor results on my time series predictions with a LSTM network. I'm looking for any ideas to improve the model.

The above graph shows the True Data vs. Predictions. The True Data is smooth zig zag shaped, from 0 to 1. However the predictions rarely reach 0 or 1.

The distribution in the prediction data-set rarely reaches 0 or 1 and it's centered around 0.5.

However the distributions in the True Data set is evenly distributed.

Here is the LSTM model built in keras:

model = Sequential()
model.add(Dropout(0.4, input_shape=(train_input_data_NN.shape[1], train_input_data_NN.shape[2])))
model.add(Bidirectional(LSTM(30, dropout=0.4, return_sequences=False, recurrent_dropout=0.4), input_shape=(train_input_data_NN.shape[1], train_input_data_NN.shape[2])))
model.add(Dense(1))
model.compile(loss='mae', optimizer='adam')

How do I get the predictions to be more similar to the true data?

I have just used LSTM to train a model predict time series value and get good result as below:

# reshape input to be [samples, time steps, features]
trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))

model = Sequential()
model.add(LSTM(40, input_shape=(trainX.shape[1], trainX.shape[2])))
model.add(Dense(1))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(trainX, trainY, epochs=10, batch_size=1, verbose=0)

Here is my result: blue is real value, orange is predict.

Ok it seems i calculated the outputs wrongly. It was not calculated fairly across the entire data-set.

I am getting better results after improving the output calculations:

It still can be improved, but it's a great start.


Add Custom Labels to NLTK Information Extractor I am working on an information extractor specifically purposed with parsing relationships between entities such as movies, directors, and actors. NLTK appears to provide the necessary tools to construct such a system. However, it is not clear how one would go about adding custom labels (e.g. actor, director, movie title).

Similarly, Chapter 7 of the NLTK Book discusses information extraction using a named entity recognizer, but it glosses over labeling details.

So, I have two questions:

How would I add custom labels?
If I have bare lists of relevant named entities (e.g. movies, actors, etc.), how can I include them as features? It appears that I would need to use IOB format, but I am unsure about how to do this when I only have lists of named entities.

Once you have your own lists of named entities, and you're only interested in extracting the relations, I believe there are simpler solutions (although I never tried relation extraction in NLTK, so I might be wrong):

ReVerb - a tool written in Java. Once it produces the results, you can simply keep the rows, where your labels are present as objects of the relation.
OpenIE - the successor of ReVerb (also written in Java). The authors claim better perfomance, and the output might be more informative.
IEPY - a relation extraction tool in Python. You should be able to provide your own labels/named entities using gazetees.
MITIE - this library has bindings in Python, and it offers relation extraction functionality.

Lots of keyword extraction techniques are there depends on factors like:
Grammatical quality of text.
Length of text
Are you looking for a single keyword or phrasal keyword etc.
But in general If you have long text and you want to extract keywords automatically from that, I would recommend you to go through follow articles:

TextRank

Rake [Rapid Automatic Keyword Extraction]

Topica

Also to extract custom(special) keywords which is not coming through above techniques, have a look at this post.


What is the reason behind taking log transformation of few continuous variables? I have been doing a classification problem and I have read many people's code and tutorials. One thing I've noticed is that many people take np.log or log of continuous variable like loan_amount or applicant_income etc.

I just want to understand the reason behind it. Does it help improve our model prediction accuracy. Is it mandatory? or Is there any logic behind it?

Please provide some explanation if possible. Thank you.

This is done when the variables span several orders of magnitude. Income is a typical example: its distribution is "power law", meaning that the vast majority of incomes are small and very few are big.

This type of "fat tailed" distribution is studied in logarithmic scale because of the mathematical properties of the logarithm:

$$log(x^n)= n log(x)$$

which implies

$$log(10^4) = 4 * log(10)$$

and

$$log(10^3) = 3 * log(10)$$

which transforms a huge difference $$ 10^4 - 10^3 $$ in a smaller one $$ 4 - 3 $$
Making the values comparable.

I'd say the main reason is not distributional but rather because of the non linear relationship. Logs often capture saturating relationships...


Check similarity between time series I have time series of parameters A, B, C and D. All of them are under influence of the same major conditions, but each one has minor differences. They are placed in different locations, A, B, C are in local1 and D is in local2.

I would like to know which one (A, B, C) has the major similarity to D. How should I approach this issue?

Sounds like a job for Dynamic Time Warping, there are implementations in Python and R.

I wrote this tutorial a while back to precisely provide guidance on these issues.
It covers four ways to quantify similarity (synchrony) between time series data using Pearson correlation, time-lagged cross correlation, dynamic time warping (as mentioned earlier), and instantaneous phase synchrony. What you choose to use will depend on how you define similarity and the characteristics of your data.


How to order a python dataframe by adding the row values? I have the following dataframe:

    M1  M2  M4   M5
N1  45  46  54   57
N2  32  36  29   56
N3  56  44  40   55
N4  57  43  42   54

How is it possible to order it according to the sum of its lines, from the largest sum to the lowest? That is, N1 in the first line, because the sum of the line values is the largest, then N4, N3 and finally N2 (because the sum of the values is the smallest). Thus:

    M1  M2  M4   M5
N1  45  46  54   57
N4  57  43  42   54
N3  56  44  40   55
N2  32  36  29   56

#create data
DF=pd.DataFrame(columns=["M1", "M2", "M3", "M4"])
DF.loc[0]=[45, 46, 54 , 57]
DF.loc[1]=[32 ,36 ,29 ,56]
DF.loc[2]=[56,44 ,40 ,55]
DF.loc[3]=[57 ,43 ,42 ,54]
DF=DF.astype(int)

#main function

DF["sum_"]=DF.apply(lambda x: sum(x), axis=1)
DF.sort_values("sum_", ascending=False)


How to make smaller categories with factor character variables I have this data set with consist of ISO3166 Alpha-2 codes for countries.
Example: DE, AD, AE etc
They are coded as factor variables in R and there are about 173 observations.

Now because there are too many and this would just overwhelm a boxplot, I want to make a contingency table with other variables by condensing the codes and create shorter categories (also coded as factors) with the codes, for example, having

DE, RE, ED, FR->  Europe

CA, US->          North America

VF, HG, HY, TY->  South America

HG, TY, UT,FT ->    Africa

How can I do that because I tried a few things that did not work.

Thank you!

P.S  These codes are made up codes but it is just to illustrate.

While you are trying this in R, I have a solution in Pyhton. You could use a similar logic in R:

# Say this is your list:
countries = ['IN', 'DE', 'US', 'UK']

# Create a list of countries in different continents like this:
asia = ['IN', 'NP']
america = ['US', 'CD']
europe = ['DE', 'UK', 'FR']

# You can then map your list with these values using a definition:
for i in range(len(countries)):
    if countries[i] in asia:
        countries[i] = 'Asia'
    elif countries[i] in america:
        countries[i] = 'America'
    elif countries[i] in europe:
        countries[i] = 'Europe'
    else:
        countries[i] = 'Others'

# Now check your list
countries
['Asia', 'Europe', 'America', 'Europe']

Once this list is ready, you could create your boxplot.

Hope this helps.


What is the meaning of likelihood? I am studying Bayes probability applied to machine learning, and I have encoutered the concept of likelihood, which I don't understand.

I have seen that the Bayes rule is:

$P(A|B)=\frac{P(B|A)P(A)}{P(B)}$

where $P(B|A)P(A)$ is the conditional probability, while $P(B|A)$ is the likelihood, but I don't understand what the likelihood is.

Can somebody please help me?

All of the answers here, including the accepted one, are conspicuously confused. I down-voted the accepted answer but downvotes of users who lack reputation in this "community" are not counted. I have a reputation of more than 200,000 (two-hundred-thousand) on math.stackexchange.com and I have a Ph.D. in statistics, but none of that counts here.

The question says $P(B\mid A)$ is the likelihood, but it ought to say instead that $A\mapsto P(B\mid A)$ is the likelihood. I.e. it's this probability as a function of $A,$ not of $B.$

It also says $P(B\mid A)P(A)$ is the conditional probability. That is not right. $P(B\mid A)$ is a conditional probability. $P(B\mid A)P(A)$ is $P(A\ \&\ B).$

Now suppose you have
$$
\Pr(A_1) = \frac 1 {20}, \quad \Pr(A_2) = \frac 2 {20}, \quad \Pr(A_3) = \frac{17}{20}
$$
and $A_1,A_2,A_3$ are mutually exclusive. This is a prior probability distribution.

Further suppose that
$$
\Pr(B\mid A_1) = \frac 9 {10}, \quad \Pr(B\mid A_2)= \frac 2 3, \quad \Pr(B\mid A_3) = \frac 1 2.
$$
Note well:

These three probabilities do not add up to $1.$ What is expressed here is not a probability distribution.
It is a function of $\text{“}A\text{''},$ a variable whose value may be either $A_1,$ $A_2,$ or $A_3.$
The likelihood is $\Pr(B\mid A)$ as a function of $A,$ not of $B.$

Bayes's theorem says: When one multiplies (pointwise) the prior probability distribution by the the likelihood and then normalizes, one gets the posterior probability distribution, i.e. the distribution conditional on the data. The "data" is the observed event $B.$

Thus
\begin{align}
& \left( \tfrac 1 {20}, \tfrac 2 {20}, \tfrac{17}{20} \right) \times\left( \tfrac 9{10}, \tfrac 2 3, \tfrac 1 2 \right) \\[8pt]
= {} & \left( \tfrac 9 {200}, \tfrac 2 {60}, \tfrac{17}{40} \right) \\[8pt]
\propto {} & \left( 27, 20, 255 \right) \\
& \text{(Here I multiplied all three components} \\
& \phantom{(} \text{by 600, which is the l.c.m. of the denominators.)} \\[8pt]
\propto {} & \left( \tfrac{27}{302}, \tfrac{20}{302}, \tfrac{255}{302} \right) \\
& \text{(Here I normalized, i.e. divided by} \\
& \phantom{(} 27+20+255 = 302 \text{ so that the} \\
& \phantom{(} \text{sum of the three components is $1.$)} \\[8pt]
= {} & \left( \Pr(A_1\mid B), \Pr(A_2\mid B), \Pr(A_3\mid B) \right) \\[8pt]
= {} & \textbf{the posterior probability distribution.}
\end{align}

Often one sees something like
$$
X\sim\operatorname{Binomial}(4,p),
$$
so that, for example
$$
\Pr(X=2) = \binom 4 2 p^2 (1-p)^{4-2}.
$$
Then if the data consists of the observation that $X=2,$ then the likelihood function is
$$
L(p) = \binom 4 2 p^2(1-p)^{4-2}.
$$
This is a function of $p,$ not of a variable whose values can be the possible values of $X,$ which in this case are $0,1,2,3,4.$

Theory
Let us use notations that are less confusing than in your question: the likelihood of a probability distribution $P$ indexed by hyperparameters $\theta$ (namely, $P_\theta$), given a set of observations ${x_i}$, is the probability that the observations were yielded by  this distribution, given $\theta$. This translates in:
$$\mathcal{L}(\theta \mid x) = P_\theta(X_1=x_1, \ldots, X_n = x_n)$$
which can also be written: $P(X = x\mid\theta)$
The link with Bayes statistics is the reversal of conditions: $\theta\mid x$  into $x\mid\theta$.
The highest the probability, the more likely it is that the observations were yielded by this distribution.
Example
Consider a set of 10 heads/tails draws, which result in 7 heads and 3 tails. You can define the likelihood that the coin is balanced ($p_\text{heads} = p_\text{tails} = 0.5$), given the results, by:
$$\mathcal{L}_\text{balanced} = P(\text{7 heads, 3 tails} \mid p_\text{heads} = 0.5) = \binom{10}{7}\left(\frac{1}{2}\right)^7\left(\frac{1}{2}\right)^3 \sim 0.117$$
And the likelihood that the probability of heads is 0.8 is:
$$\mathcal{L}_\text{balanced} = P(\text{7 heads, 3 tails} \mid p_\text{heads} = 0.8) = \binom{10}{7}0.8^70.2^3 \sim 0.203$$
From this, you can tell that it is more likely that the coin has 80% probability of heads, than it is balanced.


How to handle addresses of the restaurants to feed the data-set in the ML model? I have data from different restaurants which have also address of the restaurants now I want to predict the food delivery timing based on the given data, now the restaurant address is one of the crucial data which I need to predict the food delivery timing, now my problem is the address is in string format so how do I handle this address to feed the data in the ML model. I have a total of 35 unique addresses if I do one-hot encoding my dataset will be very large and it will take too much time to train, is there anyway except one-hot encoding to handle the addresses of restaurants.

Data Sample-

You can convert each address to a pair of geographical coordinates (latitude, longitude). In this way, you'll have a rough measure of the distance between different location. This can be done with Google Maps, for example.

I suggest you to use compute Manhattan distance between locations, in order to properly estimate travel times within a city.

Geocode !

There are API's with decent limits to free daily calls using which you can convert each individual address in the world to a ( Lat , Lon ) point. These Lat - Lon points can then be used to either do some geographical analysis or otherwise as suggested above , to find travel distances using Manhattan distance to use in an ML Model. The lat - Lon can also help you develop many interesting Insightsa dn also correlate to other Datasets.

You could also group your adresses into Primary City Regions that you can then use to estimate delivery times.

Sharing names of a few Geocoding API's

Map My India

Google Maps

Hope this works for you !


Differences in scoring from PMML model on different platforms I've built a toy Random Forest model in R (using the German Credit dataset from the caret package), exported it in PMML 4.0 and deployed onto Hadoop, using the Cascading Pattern library.

I've run into an issue where Cascading Pattern scores the same data differently (in a binary classification problem) than the same model in R. Out of 200 observations, 2 are scored differently.

Why is this? Could it be due to a difference in the implementation of Random Forests?

The difference was, it appears, due to the different implementation of Random Forests in R and Cascading Pattern (as well as openscoring which I tried later) with respect to ties in the tree voting - i.e. when an even number of trees are built (say, 500) and exactly half classify an application as Good, and the other half as Bad, the handling of those situations differs. Solved it by growing and odd (501) number of trees.

Decision Trees are unstable learners and very sensitive to changes in the input parameters.


Estimating box size from the contents I'm currently on week 4 of my Coursera course on ML, so I have much to learn about data science. However, I got the opportunity to apply what I've learned at work, and I'd like some guidance. Our company ships random objects to customers in boxes. We'd like to be able to estimate how big boxes will be, given the random objects inside.

Here's an example of the input data:

box # | contents                          | box size
----- | --------------------------------- | ---------
1     | a widget, a doodad, and a trinket | 20x12x8
2     | 3 widgets                         | 12x12x12

However, our list of items has a long tail. I did a count of total items shipped by object type, ordered by count descending. Here's the result:

rank | object count | object type
---- | ------------ | -----------
1    | 500,000      | doodad
2    | 350,000      | trinket
3    | 300,000      | widget
---  | snip         | ---
50   | 6,000        | whatyoumacallits
---  | snip         | ---
300  | 5            | quarts of blinker fluid

Etc. By item number 340, the count is 1, and there are 360 distinct items. I think one way to approach this at first would be to only consider the top 50 items, and try to do a simple polynomial regression with 50 features to estimate L, W, and H (assuming each variable is less than the previous one).

It won't be 100% accurate, but it will be better than wild guesses. But is there a better way to do this? Any advice is much appreciated.

This is known as "3D Bin Packing Problem" in literature.

https://en.wikipedia.org/wiki/Bin_packing_problem

Since this is NP-Hard; Some of the approaches  are :

Heuristics : https://www.researchgate.net/publication/226249396_A_New_Heuristic_Algorithm_for_the_3D_Bin_Packing_Problem
Deep Reinforcement Learning : https://arxiv.org/abs/1708.05930
Ensemble : https://medium.com/@alitech_2017/alibabas-ai-solution-for-the-3d-bin-packing-problem-3ce66d730ecc


Feature usage for machine learning algorithm Given a list of software installed by users as features, e.g.,

Microsoft_VC80_DebugCRT_x86_x64 1.0.0; Microsoft_VC80_DebugCRT_x86 1.0.0; ;Windows UPnP Browser 0.1.01;Adobe Acrobat Professional 10;

I want to predict whether the student will buy certain product.

Now the question is: what are the ways to turn the list of software into something learnable by a machine learning algorithm?

I would build a logistic regression with multiple independent variables. I don't think this is the only possibility, but logistic regression makes sense you are trying to model for the probability of purchase.

Obviously, your dependent variable will be whether a product is purchased, so it should be binary. Your independent variables will also be binary. You can also merge the binary variables into smaller number of categorical variables.

You should do do at least pairwise correlation (e.g. Phi coefficient). You can use it to merge highly correlated variables, such as, Microsoft Windows Word and Microsoft Windows Excel into Microsoft Office.

Your problem fits the domain of recommendation engines.
Based on user's used software, you wish to evaluate his chance of using another one(and maybe buying it).

This document describes the logic pretty well.

The item profiling phase can be done using a pairwise correlation, and other heuristics.

Be aware that usually these method need more data than standard classifiers(logistic regression/decision trees/~SVM)


Is it possible to use word2Vec to derive hyperonymy (hyponymy or ISA relation)? It's easy to have hyperonymy in WordNet, e.g. to know that "tea" is a case of "beverage". Is it possible to use word2Vec in this way?

A paper entitled "Dependency-Based Word Embeddings" found that word embeddings that defined context using dependency-based parsing generated cohyponyms. For example, "florida" was embedding near  "texas", "louisiana", "georgia", "california", and "carolina".


Derivates with respect to a vector Suppose I have an equation, $f = X^TY + \dots$ (a few more terms), where $X$ is a vector and $Y$ is a matrix of appropriate dimensions,
I want to know how can we take the derivative of $f \text{ w.r.t. } X$? I understand differentiation w.r.t one variable, but how does differentiation of another vector/matrix w.r.t a vector work?

Differentiating a function w.r.t a variable gives us the rate at with the function changes when we change the variable by a small amount. What does differentiating w.r.t a vector signify?

$f =X^T Y$ looks like this
\begin{equation}
 f= \begin{pmatrix}
  \sum_i x_iy_{i1}\\
  \sum_i x_iy_{i2}\\
  \vdots\\
  \sum_i x_iy_{in}\\
 \end{pmatrix}
\end{equation}
where $x_i$ is the $i^{th}$ element of $X$ and $y_{ij}$ is the $(i,j)^{th}$ element of $Y$.
Now, to get the gradient w.r.t $X$, is equivalent to deriving each element of $f$ w.r.t each element of $X$. This will lead to a 2D matrix:
\begin{equation}
 \nabla_X f =
 \begin{pmatrix}
  \frac{\partial}{\partial x_1} f_1 & \ldots & \frac{\partial}{\partial x_n} f_1\\
  \vdots & \vdots & \vdots \\
  \frac{\partial}{\partial x_1} f_n & \ldots & \frac{\partial}{\partial x_n} f_n
 \end{pmatrix}
\end{equation}
We can therefore easily see that
\begin{equation}
\frac{\partial}{\partial x_j} f_i
=
y_{ij}
\end{equation}
Therefore $\nabla_X f = Y$.

Derivative of a univariate vector is the same as sum of derivatives of its component(Addition rule for differentiation).

As for an extra $Y$ matrix in multiplication with $X^T$, the derivative of $f$ is calculated using partial derivative rule.

Please see the link here.


CART algorithm (Classification and regression trees) question So this is taken from an exam I just did. I'd like to know if there are any instances same as in the image where the CART algorithm could use a negative alpha and thus encourage a larger tree? Or does the algorithm state that alpha must be a non negative integer at all times?

Making an additional split will always decrease $Err$ (until pure leaves are reached), so any negative $\alpha$ will have the same minimizing $T$ as $\alpha=0$.  So we might as well take $\alpha\geq0$.  (But there is no requirement that $\alpha$ be an integer.)

Erm, unless you implement things so that even a pure node can be split, in which case a negative $\alpha$ will make the tree split even these, until every leaf consists of a single sample (and duplicates thereof) or the maximum depth is reached.

is a parameter used for regularization which is controlling the tree from being overfit , an inherent issue with CART algorithms. A negative  would lead to a larger tree which may accentuate the overfitting issue


What is dimensionality reduction? What is the difference between feature selection and extraction? From wikipedia:

dimensionality reduction or dimension reduction is the process of
reducing the number of random variables under consideration, and
can be divided into feature selection and feature extraction.

What is the difference between feature selection and feature extraction?
What is an example of dimensionality reduction in a Natural Language Processing task?

Simply put:

feature selection: you select a subset of the original feature set; while
feature extraction: you build a new set of features from the original feature set.

Examples of feature extraction: extraction of contours in images, extraction of digrams from a text, extraction of phonemes from recording of spoken text, etc.

Feature extraction involves a transformation of the features, which often is not reversible because some information is lost in the process of dimensionality reduction.

For example...if u have an agricultural land then selecting one particular area of that land would be feature selection.If u aim to find the affected plants in that area den u need to observe each plant based on a particular feature that is common in each plant so as to find the abnormalities...for this u would be considering feature extraction.In this example the original agricultural land corresponds to Dimensionality reduction.


scikit-learn - Should I fit model with TF or TF-IDF? I am trying to find out the best way to fit different probabilistic models (like Latent Dirichlet Allocation, Non-negative Matrix Factorization, etc) on sklearn (Python).

Looking at the example in the sklearn documentation, I was wondering why the LDA model is fit on a TF array, while the NMF model is fit on a TF-IDF array. Is there a precise reason for this choice?

Here is the example: http://scikit-learn.org/stable/auto_examples/applications/topics_extraction_with_nmf_lda.html#sphx-glr-auto-examples-applications-topics-extraction-with-nmf-lda-py

Also, any tips about how to find the best parameters (number of iterations, number of topics...) for fitting my models is well accepted.

Thank you in advance.

In the word sampling steps in LDA the word count is used as weights for the multinomial dist. Re-weighting the TF's by its IDF's would dispropotionally increase the chance of rare words being sampled, making them have a stronger influence in topic assignment. But rare words have equally low probabilities of showing up in all documents and topics, hence flattening out the topic distribution of each document.

That is a similar reason why it is recommended to remove stop-words before running LDA.

It is important to know that you can get the values of for the parameters you are seeking by using grid search and cross-validation. Take a look at the documentation on Scikit Learn for how to find the best parameters for fitting models and determine if it is helpful.


value saveAsTextFile is not a member of org.apache.spark.sql.DataFrame How Can I save a DataFrame as a CSV in HDFS?

I'm trying with this:

df.groupBy("Product_ID").agg(collect_list($"Stock")).saveAsTextFile("PATH/results.csv")

This is not standard part of the API of DataFrames. You can either map it to a RDD, join the row entries to a string and save that or the more flexible way is to use the DataBricks spark-csv package that can be found here.

If it's just one column you can map it to a RDD and just call .saveAsTextFile(filename)

df.rdd.saveAsTextFile("xyz") is the right answer


Linear Discriminant Analysis (LDA) before or after k-fold cross-validation? I have features extracted from a small dataset, would like to reduce the dimensions by using LDA. Also want to do a SVM classification with k-fold cross-validation.

My question is: What would be the best practice: to do LDA before the CV, or to do LDA within the CV (i.e. to each train and test fold)?

It depends on what you want to achieve.

If you want to visualize the results of your SVM classification, then you should do it after.

If you want to reduce noise, speed up training ... or whatever reason you want to reduce the dimensionality of your problem. You should do it before.

One idea here that can be useful, is to do LDA inside a pipeline and choosing the best hyperparameter within the CV.

In this example you can check a PCA and then a logistic regression. Your case will be similar but with LDA and SVM.

Sometimes k-fold is not necessary to bring better results than the standard LDA. It is preferable for large data set.


Underlying model for prediction using different prediction variables I have time-series energy consumption data for a duration of one-month. The frequency of data is half-hourly. The features of dataset are

temperature - temperature value at particular time instant
humidity -    humidity value at particular time instant
timeofday -  this corresponds to 48 half-hourly durations of a day.
weekday -  day of the week (value between 1 - 7)
prevday1 -  energy consumption at the same time-instant on previous day
prevday2 - energy consumption at the same time-instant on a day before the  previous day
prev_instant1 - energy consumption at the previous time instant. For example, for 7 pm, previous_instant1 is 6:30 pm
prev_2_hour - average energy consumption during the last two hours

Using these variable I need to forecast power (energy) consumption. The pairs plot of these variables is

I started with a linear model, but it seems that the model is performing very badly. I want to know is there any automatic technique, which can check all possible combinations of predictor variables and output the best fitted model. I can check all models (linear combinations, non-linear, splines etc) manually, but it will take huge time.

UPDATE

I am forecasting in real-time at a half-hourly rate (very short term forecasting). Also, I use separate models corresponding to different half-hours of the day. For example, In a day we have 48 half-hours so corresponding to each half-hour I have a separate model. For the whole process, I use following approach:

Create a training data set with above mentioned features. This training data set is of one month duration (30 days)
In the next step, I forecast for the next day (testing day) using specific models for each of 48 half-hour durations
At the end of the day, I retrain all my 48 models with recently updated data (recent historical 30 days)

Steps 2 and 3 continue till stopping condition. Stopping condition used is the no. of testing days. A screenshot of my training data is as:

head(training_data)

tail(training_data)

Have had the same problem before.  Used Weka tool (http://www.cs.waikato.ac.nz/ml/weka/).

The approach:
• convert time of day into a nominal (non-numeric), eg bins numbered 0-47
• convert day of week into a nominal (non-numeric)
• use M5Tree to construct a model

The rules in the tree will pop out things like "if 0 < bin < 5:30am_bin" then ... which discriminates start of day

It is better with more than 1 month of data.  There will only be 4 samples for certain days of the week, for example.  So more months would be better. And in such case again converting month to a nominal value (non-numeric) lets the rules pick up seasonal and monthly changes.


Time series - is it necessary to retrain the model when new time series data is present Say you're building a sales prediction model to predict tomorrow's sales value, as well as the next 2 weeks of daily sales. The model is being trained using daily data for the previous 1.5 years, and it follows a strong weekly seasonality pattern.

Obviously once you are happy with model performance, would it be necessary to re-train the model everyday to capture data up to and including yesterday in order to get the most accurate prediction for tomorrow's sales value? Essentially, the model would be trained on a rolling 1.5 year data set to capture yesterday's sales value.

Or does it depend on the type of model being used? Whether you need to to completely re-train when new data is present

I can see why it would make sense to re-train your standard time series forecasting models (ARIMA etc.), but I can also understand more sophisticated models (Neural Networks etc.) may generalize well enough to not have to re-train everyday.

I am looking for an explanation of models where you would and wouldn't re-train when new time series data is present.

It is good practice to retrain model once in a while. That "in a while" depends on the specific task, of course. This is due to the fact that countless unmeasurable things change continuously.

Think for example about a model to forecast companies' revenues: the rules of the game change continuously, the market today is not like the market one year ago, not to mention five years ago. This is the first example that I thought about, but it's true for pretty much any possible application of ML.

That is why it was observed that models' performance deteriorates in time. For that reason, it is good practice to keep your models updated with fresh data.

You need to retrain your model every time you want to generate a new prediction:

For most ML models, you train a model, test it, retrain it if necessary until you’ve gotten satisfactory results, and then evaluate it on a hold out data set. After you’re satisfied with the model’s performance, you then deploy it to production. Once in production, you score new data as it comes in. Eventually after a few months, you might want to update your model if a significant amount of new training data comes in. Model training is a one time activity, or done at most at periodic intervals to maintain the model’s performance to take int account new information. For instance, the visual properties of cats are stable over time. We don’t expect cats to look different next week, or next year, or even ten years from now. Given enough data, the model we trained this week is good enough for the foreseeable future as well.

For time series models, this is not the case. Instead we have to retrain our model every time we want to generate a new forecast. To understand why this happens consider this...trends, and related seasonal variations, change over time!!


Damerau-Levenshtein Edit Distance in Python I found some python codes on Damerau Levensthein edit distance through google, but when i look at their comments, many said that the algorithms were incorrect. I'm confused.

Can someone share a correct python code on Damerau Levensthein Distance?

Thank you.

Which one is correct? (Sorry, i don't know how to properly edit the typing with my handphone, my only online device for the time being)

https://www.guyrutenberg.com/2008/12/15/damerau-levenshtein-distance-in-python/

The first one,

"""Compute the Damerau-Levenshtein distance between two given strings (s1 and s2)"""

    def damerau_levenshtein_distance(s1, s2):

        d = {}
        lenstr1 = len(s1)
        lenstr2 = len(s2)
        for i in xrange(-1,lenstr1+1):
            d[(i,-1)] = i+1
        for j in xrange(-1,lenstr2+1):
            d[(-1,j)] = j+1

        for i in xrange(lenstr1):
            for j in xrange(lenstr2):
                if s1[i] == s2[j]:
                    cost = 0
                else:
                    cost = 1
                d[(i,j)] = min(
                               d[(i-1,j)] + 1, # deletion
                               d[(i,j-1)] + 1, # insertion
                               d[(i-1,j-1)] + cost, # substitution
                              )
                if i and j and s1[i]==s2[j-1] and s1[i-1] == s2[j]:
                    d[(i,j)] = min (d[(i,j)], d[i-2,j-2] + cost) # transposition

        return d[lenstr1-1,lenstr2-1]

"""The second code is listed below"""

https://gist.github.com/pombredanne/0d83ad58f45986ddeb0917266e106be0

Damerau-Levenshtein edit distane implementation

Based on pseudocode from Wikipedia: https://en.wikipedia.org/wiki/Damerau-Levenshtein_distance

Possible improvement by treating 1 addition + 1 deletion = 1 substitution
between transposed characters:

Damerau-Levenshtein distance for "abcdef" and "abcfad" = 3:

substitute "d" for "f"
substitute "e" for "a"
substitute "f" for "d"

Or alternatively:

transpose "d" and "f"
delete "a"
insert "e"

It's obvious that (2) and (3) in the second analysis are really just one
substitution:

transpose "d" and "f"
substitute "e" for "a"

With this variant, the distance between "abcdef" and "abcfad" is in fact 2.

def damerau_levenshtein_distance_improved(a, b):

    # "Infinity" -- greater than maximum possible edit distance
    # Used to prevent transpositions for first characters

    INF = len(a) + len(b)

    # Matrix: (M + 2) x (N + 2)
    matrix  = [[INF for n in xrange(len(b) + 2)]]
    matrix += [[INF] + range(len(b) + 1)]
    matrix += [[INF, m] + [0] * len(b) for m in xrange(1, len(a) + 1)]

    # Holds last row each element was encountered: DA in the Wikipedia pseudocode
    last_row = {}

    # Fill in costs
    for row in xrange(1, len(a) + 1):
        # Current character in a
        ch_a = a[row-1]

        # Column of last match on this row: DB in pseudocode
        last_match_col = 0

        for col in xrange(1, len(b) + 1):
            # Current character in b
            ch_b = b[col-1]

            # Last row with matching character
            last_matching_row = last_row.get(ch_b, 0)

            # Cost of substitution
            cost = 0 if ch_a == ch_b else 1

            # Compute substring distance
            matrix[row+1][col+1] = min(
                matrix[row][col] + cost, # Substitution
                matrix[row+1][col] + 1,  # Addition
                matrix[row][col+1] + 1,  # Deletion

                # Transposition
                # Start by reverting to cost before transposition
                matrix[last_matching_row][last_match_col]
                    # Cost of letters between transposed letters
                    # 1 addition + 1 deletion = 1 substitution
                    + max((row - last_matching_row - 1),
                          (col - last_match_col - 1))
                    # Cost of the transposition itself
                    + 1)

            # If there was a match, update last_match_col
            if cost == 0:
                last_match_col = col

        # Update last row for current character
        last_row[ch_a] = row

    # Return last element
    return matrix[-1][-1]


In K-means, what happens if a centroid is never the closest to any point? I am implementing K-Means from scratch and that exercise raised a question.

To update my centroids, for each centroid, I have to find the points for which that centroid is the closest.

In some cases, especially when the number of centroids is high and the number of instances is low (i.e. k=20 and 100 instances), I find centroids for which no point has them as their closest centroid. In other words, they become "orphans" as no instances are allocated to them.

In the example above, the two centroids (X) at the top clearly both have points for which they are the closest centroid. But the centroid at the bottom is never the closest centroid for any instance.

How do I deal with this?

Should the lone centroid remain unmoved?
Should I move that centroid? If yes, how?
Should I remove it?

Is there a standard way to deal with this?

------------ EDIT: --------------

On Github, I found this:

https://github.com/klebenowm/cs540/blob/master/hw1/KMeans.java

This suggests that if a centroid becomes an "orphan", it should be assigned to the point that is the furthest from its centroid.

This seems like a sound method, is there any paper or theory supporting this?

You should leave the lone centroid unmoved. In the next iteration, it's possible that the cluster centers have moved in such a way that now there are some instances that are closest to the lone centroid, and it can get picked up. At the end of the k-means algorithm, you could remove clusters that have no instances associated with them, but it doesn't really matter.

At the end of the day, it's probably not a good idea to use k-means in the manner that you described, and we should not expect it necessarily to produce good results.

You don't have to worry about it, as we know, in k-means clustering, you only have to choose the initial centroids. This would create first iteration of clusters. In the next iteration, the centroids would move to the center of the newly created clusters. this whole process will continue till you get convergence.
Now, the number of centroids selected at the first place are good or not, is a separate question.
Also, we also know, the result of a k-means is dependent upon the initial selected centroids and we have various methods of selecting intital centroids, but just answer your question here, run the full algorithm and you wont have such lone centroids at the end of the algo.


GBM vs XGBOOST? Key differences? I am trying to understand the key differences between GBM and XGBOOST. I tried to google it, but could not find any good answers explaining the differences between the two algorithms and why xgboost almost always performs better than GBM. What makes XGBOOST so fast?

Quote from the author of xgboost:

Both xgboost and gbm follows the principle of gradient boosting.  There are however, the difference in modeling details. Specifically,  xgboost used a more regularized model formalization to control over-fitting, which gives it better performance.
We have updated a comprehensive tutorial on introduction to the model, which you might want to take a look at. Introduction to Boosted Trees
The name xgboost, though, actually refers to the engineering goal to push the limit of computations resources for boosted tree algorithms. Which is the reason why many people use xgboost. For model, it might be more suitable to be called as regularized gradient boosting.

Edit: There's a detailed guide of xgboost which shows more differences.
References
https://www.quora.com/What-is-the-difference-between-the-R-gbm-gradient-boosting-machine-and-xgboost-extreme-gradient-boosting
https://xgboost.readthedocs.io/en/latest/tutorials/model.html

XGBoost implementation is buggy. Crashed silently when training on GPU on v 082 . It happened to me as well on v 0.90, so the issue has not been addressed so far, and the "fix" provided in GitHub didn't work for me.

LGBM 2.3.1 works like a charm out of the box, though installing it requires a little more effort. So far no issues training on GPU.

About XGBoost being "so fast", you should take a look at these benchmarks.


Python vs R for machine learning I'm just starting to develop a machine learning application for academic purposes. I'm currently using R and training myself in it. However, in a lot of places, I have seen people using Python.

What are people using in academia and industry, and what is the recommendation?

Some real important differences to consider when you are choosing R or Python over one another:

Machine Learning has 2 phases. Model Building and Prediction phase. Typically, model building is performed as a batch process and predictions are done realtime. The model building process is a compute intensive process while the prediction happens in a jiffy. Therefore, performance of an algorithm in Python or R doesn't really affect the turn-around time of the user. Python 1, R 1.
Production: The real difference between Python and R comes in being production ready. Python, as such is a full fledged programming language and many organisations use it in their production systems. R is a statistical programming software favoured by many academia and due to the rise in data science and availability of libraries and being open source, the industry has started using R. Many of these organisations have their production systems either in Java, C++, C#, Python etc. So, ideally they would like to have the prediction system in the same language to reduce the latency and maintenance issues.
Python 2, R 1.
Libraries: Both the languages have enormous and reliable libraries. R has over 5000 libraries catering to many domains while Python has some incredible packages like Pandas, NumPy, SciPy, Scikit Learn, Matplotlib. Python 3, R 2.
Development: Both the language are interpreted languages. Many say that python is easy to learn, it's almost like reading english (to put it on a lighter note) but R requires more initial studying effort. Also, both of them have good IDEs (Spyder etc for Python and RStudio for R). Python 4, R 2.
Speed: R software initially had problems with large computations (say, like nxn matrix multiplications). But, this issue is addressed with the introduction of R by Revolution Analytics. They have re-written computation intensive operations in C which is blazingly fast. Python being a high level language is relatively slow. Python 4, R 3.
Visualizations: In data science, we frequently tend to plot data to showcase patterns to users. Therefore, visualisations become an important criteria in choosing a software and R completely kills Python in this regard. Thanks to Hadley Wickham for an incredible ggplot2 package. R wins hands down. Python 4, R 4.
Dealing with Big Data: One of the constraints of R is it stores the data in system memory (RAM). So, RAM capacity becomes a constraint when you are handling Big Data. Python does well, but I would say, as both R and Python have HDFS connectors, leveraging Hadoop infrastructure would give substantial performance improvement. So, Python 5, R 5.

So, both the languages are equally good. Therefore, depending upon your domain and the place you work, you have to smartly choose the right language. The technology world usually prefers using a single language. Business users (marketing analytics, retail analytics) usually go with statistical programming languages like R, since they frequently do quick prototyping and build visualisations (which is faster done in R than Python).

I do not think Python has point-click GUI that turn it into SPSS and SAS. Playing around with those is genuinely fun.


Binary classfication vs One-class classification Why do we need samples of both classes for the training of binary classification algorithms, if one-class algorithms can do the job with only samples from one class?

I know that one-class algorithms (like one-class svm) were proposed with the absence of negative data in mind and that they seek to find decision boundaries that separate positive samples (A) from negative ones (Not A).

Hence the traditional binary classification problem (between (A) and (B) for example) can be formulated as a classification of (A) and (not A = B).
Is it about better classification results or am I missing something?
Thank you in advance

Binary classification is needed when requirement is to capture data into two classes. if you can't capture data in two classes where you need only one you go for One-class.
you can check this link for better explanation
http://rvlasveld.github.io/blog/2013/07/12/introduction-to-one-class-support-vector-machines/

If you take binary classification, svm tries to find best possible space between A and B. If there is only one class A model tries to create a boundary around it and classify. Take for example patient disease classification: For +ve some symptoms t1, t2, t3, t4, t5 for -ve he has t1, t2, t7. in the above case it is difficult to classify using one class because model classifies patient having t1, t2 as +ve because of proximity to +ve class. The second label gives you more info for better classification.

I think this is what you are looking for:

A binary classifier is used to classify an instance into one of two classes and the reason behind using binary classifier for one class problem is that either an instance belong to that class or not. For example, if your problem is to predict whether there will be rain tomorrow. So either there will be rain tomorrow or not.
Another example is that Given an image, your classifier task to precit whether it is male or not. Though you are only concern about one class which is male but still there are two classes i.e. male or not(male).

So, when training your classifier for one class, your data should be instances of that class plus instances of not that class. So, your classifier is capable of accurately classifying an instance whether it belongs to that class or not.

If you want to know anything else, let me know. I'll be glad to help you.


Combining binary features with textual ones I have a binary feature that i want to use it with textual features i.e. unigrams. I use logistic regression and TF/IDF for representing text. So i simply add a unique feature, say ss or oo, to text of each instances. But in practice, i see adding more number of these features to instances, say two oo or ss or more get me a better results. What is the reason? How these weights improve the classification results? Should not logistic regression can get more weights to this features instead of weighting them by hand?

hstak is a good solution. Using this I combined these features.


What does "baseline" mean in the context of machine learning? What does "baseline" mean in the context of machine learning and data science?

Someone wrote me:

  Hint: An appropriate baseline will give an RMSE of approximately 200.

I don't get this. Does he mean that if my predictive model on the training data has a RMSE below 500, it's good?

  And what could be a "baseline approach"?

A baseline is the result of a very basic model/solution. You generally create a baseline and then try to make more complex solutions in order to get a better result.
If you achieve a better score than the baseline, it is good.

As we have many machine learning algorithms, we have to know which ML algorithm best suits for our problem. This will be identified by Baseline Prediction algorithm,

A baseline prediction algorithm provides a set of predictions that you can evaluate as you would any predictions for your problem, such as classification accuracy or RMSE.

The scores from these algorithms provide the required point of comparison when evaluating all other machine learning algorithms on your problem.

for further information we have a very good blog on ML : What does "baseline" mean in the context of machine learning?