fabiochiu/medium-articles huggingface dataset:
Content Marketing: Does It Increase Brand Awareness?
By: Alex Greenwood
Tags: Content Strategy, Brand Strategy, Content Writing, Content Marketing, Public Relations

Rising above the clutter in today’s challenging media environment is crucial — and difficult to attain. One of the most successful strategies is to provide what’s in high demand from editors and publishers: quality formatted content. Whether you are sharing ways to get a building permit or how to make a moist Thanksgiving turkey, there is likely an audience for it.

Photo by Kaboompics .com from Pexels

One way to get your brand in front of the appropriate audiences in a positive way is to offer interesting, relatable content that websites, newspapers, and broadcast news outlets can use. This practice is known as content marketing.

What is content marketing? It’s a simple concept, and I like this definition provided by the Content Marketing Institute:

Content marketing is a marketing technique of creating and distributing valuable, relevant and consistent content to attract and acquire a clearly defined audience — with the objective of driving profitable customer action.

In other words, if you’re reading this and sharing it with others, then content marketing is working. According to iMedia, 70 percent of marketers report that content marketing increased their brand awareness. I can tell you that fully 30% of my client prospects say they decided to contact me based on my blog or social media shares. Says a lot, doesn’t it?

Now, it’s still incumbent upon me to close the deal, but I like my odds. I love the fact that we can share or curate information I find interesting and it simultaneously becomes a method of prospecting for new business.

Content marketers can help you craft a story for your brand or product/service that appeals to editors. They then shape your message into a variety of formats, including full-page features, mat releases, product roundups, infographics, and more. Your project is distributed to networks of editors, who receive the content for free.

My clients count on me to create compelling, sharable, interesting content to attract customers. I submit that a public relations consultant or agency is the best answer, as we are experts at packaging information in a way that makes it compelling and useful to the news media and consumers.
5 Ways to Become a Better Reader
By: Tom Terado
Tags: Reading, Content Writing, Reading Books, Sapiens, Content Marketing

We read up to 3000 words a day.

We go through countless texts, emails, and books but how much information do we remember?

It’s quite a shame when we read a book but forgot the main themes of it after a few months. For this reason, here are some actionable steps you can take to becoming a better read.

Take notes

Everyone has a different style of learning but we memorize better by writing things down. Whether you read on an iPad or paperback, use a highlighter or write down small notes about the chapter you read. This will help your brain process the themes of the book.

Keeping a public or private journal will be a great reference for when you want to revisit the themes of the book. A suggestion is to write them in your Notes App or publish them on Medium / Substack.

Slow down and Speed up

Changing your pace of reading throughout a book is useful in different contexts. It allows you to identify which parts of the book to skim or slow down.

Skim: If you understand the main point, move on.

Slow down: When the argument is hard to understand. Try re-reading it again.

You don’t have to keep reading

Books should be enjoyed. Reading shouldn’t feel like a chore. There is a common belief that we should finish books from start to finish. Why waste time on a book that doesn’t bring you joy?

So don’t feel obliged to keep reading.

You should still write down why you couldn’t finish it. Was it not engaging enough? Was it too long to explain each point? This helps you reflect on what type of books engage you the most.

Allocate time

Setting allocated time to read with purpose will allow you to enjoy your reading. The more invested you are, the more you get out of it. Whether it’s first thing in the morning, during your lunch break, or before bed, set aside time to read.

How: set yourself small goals.

For example, set aside time to read for 20 minutes on a Wednesday evening. Once you do that you can increment it to 25 minutes. Setting small goals will make all the difference in your reading habits.

Reading in a comfortable environment will also help you read with more purpose and focus. So grab a nice cup of tea and read on your most comfortable chair.

Explain the book in five sentences

We learn best by teaching. So next time you bring a book up in conversation, explain the book in less than five sentences. This demonstrates that you understand the main themes. It will also make for a more engaging conversation.
How To Earn Money As A Freelancer
By: Piyush Kumar
Tags: Copywriting Tips, Online Writing, Content Writing, Freelance Writing, Freelancers

Hey my fellow freelancers, Sup? Trying to make money? I know I know. It gets frustrating.

But don’t worry, today I am going to guide you through a step by step method of how you can earn money as a freelancer.

Whether you are a writer, designer, programmer, or any other type of freelancer, these are the fundamentals of online money making.

So these are the 3 steps :

Gain trust Pitch Build a system

Let me elaborate

GAIN TRUST :

First of all, you need proof. You can’t just go out there and say “Gimme work!”

You need to work for yourself, whatever you are doing, produce some samples first.

If you have worked for your friends, you can also get their testimonials regarding your skills.

After collecting some proof, make a portfolio with it. Describe all your skills and mention all the work you’ve done.

PITCH :

Now that you have the portfolio, you go out there and make your name. Literally.

Be open about your work experience and how dedicated you are. Go tell everyone you know and say

“I am a freelancer and I am willing to work”

Not everyone in real life, tell everyone online. Those who are interested will ask you what you can do.

Give them some samples, show them your portfolio. Land some gigs do some short term/long term work for some months or for a while.

Write all of them in your portfolio and make it bigger and bigger.

BUILD A SYSTEM :

Now that you have some work experience under your belt, it’s time to become a business.

Not start a business, become a business. Now go on every single social media platform and start giving free value to people.

A lot of people will like your free value when you are experienced, and on the side you can also say

“Hey, I am also working in this field and as you have seen, I am experienced so you can hire me too”

You can do this with a website, an Instagram page, a youtube channel, a LinkedIn page or whatever social media platform that you like.

This step will take a lot of time, and some money, but when you have achieved this step, believe me, you won’t need to have a job ever again.

I am sure you liked this article, so you can check out my other articles too.

And by the way I also have a youtube channel. Funny how a person can do so much haha

https://www.youtube.com/channel/UCFw8YTRt8aNpRurOItlkgig
My Baby Brand, Shape My Content, is Now GoodFirms-Approved!
By: Neha Bhatiya
Tags: Awards, Recognition, Content, Content Writing, Content Marketing

Not long ago, sometime in June 2019 (June 10th to be precise!), I started a venture with no name. Within a week or so, I decided I’d call it Shape My Content and will make it a truly client-centric content writing agency.

From what I can recall, the initial few months were a drag. Constant hustle, critical comments from clients, messy team management, and several other concerns were blurring my dream of running a startup. In fact, there were moments when I was reconsidering my decision of dropping my well-paying job and jumping into this unknown entrepreneurship journey.

But as they say, persistence is the key.

I learned to read between the lines and focused on generating value for my clients.

From keeping tabs on market trends to building a team of like-minded go-getters, I did everything that helped me to stay put and continue what I started. And I’m glad I did - things are getting better with each passing day. Couldn’t be more grateful!

And hey, my baby brand recently got featured in GoodFirms’s list of top content writing agencies. Yes, I’m happy! Here’s what this leading research and review giant has to say about us!

Unique and Quality Content Makes Shape My Content Earn a Prominent Position at GoodFirms Content marketing helps brands elevate their brand value and website traffic. Likewise, Shape My Content (SMC) is assisting clients to achieve their marketing goals by offering them authentic, optimized, user-friendly, and SEO-friendly content that draws and sustain organic traffic. So, it comes as no surprise that the company has earned a prominent spot in the GoodFirms’ list of top writing companies. Shape My Content — Your Trusted Content Writing Partner Founded in 2019, Shape My Content is an Indian content writing agency offering the best content creation and management services. The company specializes in writing website blogs, social media posts, webpage content, editing/proof-reading, case studies, drip emails, SEO content, product description, listicles, and many more. The SMC team is proficient in curating content that enhances a website’s and brand’s reputation. Realizing that branding is much more than just making a name and logo, the team supports the brands by offering optimized content solutions for them. The content team comprising active storytellers and creative content writers crafts 100% authentic content for different clients ranging from startups to enterprises. And even crafting elaborate, provoking, sharp, cogent yet lucid copies for B2B and B2C sites for their ever-evolving content marketing requirements. Plus, they are also pro in developing, analyzing, and redeploying content materials that engage and attract wider audiences. So far, SMC has assisted around 93+ happy clients with 450+ successful projects and served in 14 countries.

Read more at https://bit.ly/3kQyUZH
Why we use the standard deviation and not median absolute deviation or other functions
By: Peter Flom
Tags: Standard Deviation, Variance, Statistics, Regression

Formula for the sample standard deviation

On Quora, someone asked about why, in the standard deviation, we square and take square roots rather than use mean absolute deviation. Here’s my answer:

We can do either, but a more common method is the median absolute deviation, rather than the mean absolute deviation (which is what you propose). The median absolute deviation is very robust to outliers. And there are other possibilities for measures of spread.

The term “variance” was introduced in a paper by Ronald Fisher in 1918. It has some nice properties.

First, Fisher used it because it is part of the Normal distribution, and many traits follow that distribution (at least approximately). The probability density function for the Normal involves the standard deviation, not the median or mean absolute deviation.

Second, the variance is analogous to the moment of inertia in physics.

Third, in regression, ordinary least squares uses the variance. And, before computers, OLS regression was much, much easier to calculate than some other kinds of regression, some of which are actually preferable, now that we have computers. For instance, quantile regression has many nice properties but was mostly unusable until fairly recently (link goes to an article I wrote).

Fourth, variances are additive (when the components are not correlated). MAD is not, nor is MAD squared. The R code below shows one example of this:
Polinomal Regresyon -Makine Öğrenmesi (2020)
Tags: Regression, Polinomal Regresyon, Polynomial Regression, Python, Machine Learning

Makine Öğrenmesinde Polinomal Regresyon ile karşınızdayım. Bu algoritma, lineer regresyon’la beraber kullanılarak modelin eğitilmesi sağlanır.

1- Veri Setini Getirme/İnceleme

import seaborn as sns

import pandas as pd

import numpy as np

from sklearn.impute import SimpleImputer



data = sns.load_dataset("tips")

df = pd.DataFrame(data)



# Eksik Verileri Ortalama ile Doldurma

imputer = SimpleImputer(missing_values=np.nan, strategy="mean")

imputer = imputer.fit(df[["total_bill"]])

df[["total_bill"]] = imputer.transform(df[["total_bill"]])



# Kategorik Olmayan Değişkenler

total_bill = df[["total_bill"]]

tip = df[["tip"]]

size = df[["size"]]



# Kategorik Değişkenler İçin 0-1 Dönüşümü

smoker = pd.get_dummies(df[["smoker"]])

time = pd.get_dummies(df[["day"]])



# Kukla Değişkeni Silme

smoker = smoker.drop("smoker_Yes", axis=1)



# Verileri Birleştirme

new_data = pd.concat([total_bill, size, smoker, time, tip], axis=1)



# Train ve Test Belirleme

training = new_data.iloc[:, 0:7]

testing = new_data.iloc[:, 7:8]



# Train Test Ayırma

from sklearn.model_selection import train_test_split



x_train, x_test, y_train, y_test = train_test_split(training, testing, test_size=0.30, random_state=42)

2- Model Eğitme

from sklearn.linear_model import LinearRegression linear_reg = LinearRegression() linear_reg.fit(x_train, y_train) from sklearn.preprocessing import PolynomialFeatures polynomial_reg = PolynomialFeatures(degree=3) xp = polynomial_reg.fit_transform(x_train) linear_reg2=LinearRegression()

linear_reg2.fit(xp, y_train)

3- İstenilen Değerlerle Tahmin

print(linear_reg2.predict(polynomial_reg.fit_transform([[16.99,2,1,0,0,0,1]])))

4- MSE (Mean Squared Error)

from sklearn.metrics import mean_squared_error



y_pred = linear_reg2.predict(polynomial_reg.fit_transform(x_test))



rmse = np.sqrt(mean_squared_error(y_test, y_pred))

print("Test RMSE: ", rmse)

5- Tahmin v Gerçek

y_pred = linear_reg2.predict(polynomial_reg.fit_transform(x_test))



y_tahmin=pd.DataFrame(data = y_pred)

y_normal = pd.DataFrame(data = y_test)

y_tahmin.index = y_normal.index



result=pd.concat([y_tahmin, y_normal],axis=1)

result.columns=[["Tahmin","Gerçek"]]



print(result)

Umarım faydalı olmuştur. Diğer algoritmalarda görüşmek üzere :)

Önceki Yazı (Çoklu Lineer Regresyon)/ Sonraki Yazı (Destek Vektör Regresyonu)
Data Science Project — The Hidden Tastes of US Zip Codes
Tags: Food, Clustering, Data Science, Regression

As most of us found ourselves at home during the covid-19 crisis I decided to use the time and learn a new subject I always wanted and didn’t found the time for it.

I have joined a course over Coursera that teaches both python and some machine learning. As the final part of this course, our task was to choose a project using location data and research an area based on that data.

I’m passionate about food, So why not try to combine both, the locations of restaurants and this project, trying to find some patterns.

The project explores different US states by zip codes; For each zip code, I have used forsquare API to retrieve information about the restaurants around. Hoping to identify a difference in the taste of the population between different states and different zip codes based on the types of restaurants.

I wanted to add some economical information on top of that but was unable to find any up-to-date data source by zip codes. So I decided to take a different approach and use house values by zip code, the data is available via Zillow.

Using the house data instead of income I’ll try and use it to identify any correlation between the price of a house and the sort of restaurants around, or maybe the sort has no correlation but the number does, both things can be tested using this data set.

This will be a three-part post, this is a summary of the work, in the next two posts I will explain the work in details with code example.
Gaussian Process Regression
By: Jamie Keng
Tags: Options, Regression, Pricing, Gaussian Process, Machine Learning

Photo by Chris Liverani on Unsplash

In this article, I would like to discuss briefly Multi-response Gaussian process(MGP), which takes correlations between responses into account when making predictions.

The intention of this article is not to dive deep into the mathematical background of multi-output Gaussian process regression but to give the audience a taste of its implementation in prediction option prices.

The following explanation was extracted from [1]. If you would like to know the mathematical background and details of multi-response Gaussian process regression, please refer to [1–2].
Predicting the Volatility of Stock Data
By: Ramji Balasubramanian
Tags: Predictions, Regression, Basics, Volatility, Stock Market

Lets Validate the observed result using a basic regression method.

How can we validate that all the features are not important to predict the volatility?. One way which I used here is, First will predict the volatility using all the columns in the data we have and then will find the important feature and use only that to predict the volatility and measure the performance using Mean Absolute Error(MAE).

Evaluation Metric

from sklearn.datasets import make_regression

from sklearn.linear_model import LinearRegression

from matplotlib import pyplot

# define dataset

X, y = make_regression(n_samples=2373, n_features=7)

# define the model

model = LinearRegression()

# fit the model

X = df.drop([‘volatility’,’Date’,’returns’],axis=1).values

y = df[‘volatility’].values

model.fit(X, y) #Mapping column names for feature importance graph

map_colums ={}

col =df.drop([‘volatility’,’Date’,’returns’],axis=1).columns

for i in range(len(col)):

map_colums[i] = col[i]

map_colums

# get importancea

importance = model.coef_

# summarize feature importance

for i,v in enumerate(importance):

print(‘Feature: ‘,map_colums[i],’Score: %.7f’ % v)

# plot feature importance

pyplot.bar([map_colums[x] for x in range(len(importance))], importance)

pyplot.show()

Feature Importance

Performance of the model using all the features

from sklearn.model_selection import train_test_split

from sklearn.metrics import mean_absolute_error

# create regressor object `

X,x_valid,y,y_valid = train_test_split(X,y,test_size=0.1)

print(X.shape,y.shape,x_valid.shape,y_valid.shape)

regressor = LinearRegression()



# fit the regressor with x and y data

regressor.fit(X, y)



pred = regressor.predict(x_valid)

mean_absolute_error(pred,y_valid)

Score: 0.12109432688323198

Performance of the model using only the selected feature(Low)

From the above feature importance graph it is very clear that Low column is best for the regression model to fit and predict the volatility. Because for regression model, the key element is the selected features should be either negatively correlated feature or positively correlated feature.

X = df[[‘Volume’,’Low’]].values

y = df[‘volatility’].values

X,x_valid,y,y_valid = train_test_split(X,y,test_size=0.1)

print(X.shape,y.shape,x_valid.shape,y_valid.shape)

regressor = LinearRegression()



# fit the regressor with x and y data

regressor.fit(X, y)

pred = regressor.predict(x_valid)

mean_absolute_error(pred,y_valid)

Score: 0.11544891600804784

From the results of two different approach, I would say second is better even thought the difference in MAE is lower. The reason behind my conclusion is instead of use 5 different features, model was able to predict the volatility with one feature and its better than later. This will save the computational time for training as well. I hope this article gives a brief idea on how to start building a basic regression model for any prediction problem.

Tomorrow will see how we can apply some Deep Learning Methods for predicting the Volatility. Keep reading !!!
Checking The Assumptions For Linear Regression
By: Datascience George
Tags: Statistics, Data Science, Regression

In this post I will talk about the assumptions for linear regression. Some must be checked before modeling begins. Others need to be checked afterward using the residuals obtained from training. It is important to check all of the assumptions for a model because failing to do so can lead to unreliable results in the real world, even if the performance metrics from evaluation on the testing set are great.

The first assumption is that the feature variables being modeled have a linear relationship with the target variable. This means that as one feature variable increases, the target variable responds by either increasing or decreasing. The second assumption is that there is little to no multicollinearity between feature variables. Multicollinearity is when the features have strong linear relationships with one another. To test linear relationships you can use scatter plots. Another way is to use a correlation heatmap. This can be done using pandas and seaborn. Pandas has a method to turn a data frame into a matrix of correlation values and seaborn has a plot that color codes the values to visualize the matrix. It is best to use absolute values of the correlation coefficient because in this case it doesn’t matter if the correlation is positive or negative. Avoid using features that have a correlation with each other higher than 0.6. Here is a code example.

import pandas as pd

import seaborn as sns # dataset

df = pd.read_csv('data.csv') # correlation matrix

matrix = abs(df.corr()) # heatmap

sns.heatmap(matrix)

If the data meets the above assumptions go ahead and model with the data. Save the residuals (errors) from the training set and use these to test the other assumptions.

The first assumption for the errors is that they are normally distributed. To check for this you can use QQ-plots or simply use a histogram. If the errors are not normal then the model’s performance can not be predicted reliably. This could be caused by bad data. Try removing outliers from the data and removing any features that are imbalanced.

The second assumption for the errors is that they have homoscedasticity. This means that the variance of the errors does not change with the value of the target variable. For instance if a regression model had a low error for predicting values that lie between 100 and 200, but had very high error for values that lie between 400 and 500, that would make the model unreliable. In order to check for this make a scatter plot and plot the residuals against the true target value. Then draw a horizontal line at zero where the errors should be centered. If the plot is a straight bar then the data has homoscedasticity. However if the data forms a cone then it does not. To fix this problem try applying transformations to different variables such as a logarithmic transformation. This can improve performance. Here is a code example and a picture showing residuals with homoscedasticity.
SVM — Support Vector Machine 🐱‍🚀
By: Sri Vigneshwar Dj
Tags: Svm, Regression Analysis, Svr, Regression, Machine Learning

SVM (Support Vector Machine) algorithm is used to find the best fit hyperplane in n dimensional space.

Here n — no. of features or independent variable (xi), SVM can be used for both regression and classification (SVR and SVC).

Why SVM?

SVM is a linear model for classification and regression problem, SVM can solve linear and non-linear data in effective way. Let’s us take non-linear data set like this 👇

Here it’s not practical to draw a line to classify or to perform regression, no other algorithm can solve it too (Linear , Logistic , ..), we can use SVM to achieve the desired results.

Let’s see HOW?

In SVM there is a concept called kernel, it helps plotting non-linear data to higher dimensional to fit the hyperplane.

In higher dimension we can fit the hyperplane, so we can able to classify the data point. After the fitting the hyperplane we can convert to original dimension.

SVM can be used for Classification and Regression

Let’s how to implement SVR in python
Understanding Simple Regression in One Story
By: Eze
Tags: Math, Regression

If you remember this equation y = mx + c (also expressed as Y = bX + a), then here’s a story to help you understand it even more; especially in terms of real world scenario. At the end, you may find it even more enjoyable and convenient to explain this important mathematical expression to your friends, family, kids, and even the general public.

For the benefit of doubt, regression is a mathematical equation that helps us understand how a change in one thing (the independent variable) leads to a change in another (dependent variable).

Let’s assume you live in a city where taxi rides are metered. Did you know that the moment you enter a taxi, you’re charged a fixed fee of about $2–5? It means that if the car didn’t move and you end the trip (by deciding not to proceed), you must pay $2.

This fixed fee is the constant, c, in y = mx + c.

Now that we’ve got this out of the way; let’s say you decide to proceed with the journey, you taxi app has a rate that calculated at $ per km. The rate is the “m” in y = mx + c, while the total distance of the ride is x. However, the actual rate calculate is more complex than this but at least you get the idea of how regression. Hope this was helpful?
Linear Regression in Python Scikit Learn
By: Mohomed Ashkar Haris
Tags: Linear, Regression, Scikitlearn, Python, Sklearn

Linear Regression in Python Scikit Learn

import numpy as np

import matplotlib.pyplot as plt

import pandas as pd

import seaborn as sb

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LinearRegression

Import Dataset

Dataset=sb.load_dataset('tips')

print(Dataset) total_bill tip sex smoker day time size

0 16.99 1.01 Female No Sun Dinner 2

1 10.34 1.66 Male No Sun Dinner 3

2 21.01 3.50 Male No Sun Dinner 3

3 23.68 3.31 Male No Sun Dinner 2

4 24.59 3.61 Female No Sun Dinner 4

.. ... ... ... ... ... ... ...

239 29.03 5.92 Male No Sat Dinner 3

240 27.18 2.00 Female Yes Sat Dinner 2

241 22.67 2.00 Male Yes Sat Dinner 2

242 17.82 1.75 Male No Sat Dinner 2

243 18.78 3.00 Female No Thur Dinner 2



[244 rows x 7 columns]

Observe Dataset

Dataset.plot(x='total_bill',y='tip',style='o')

plt.show()

Prepare Data

x = np.array(Dataset['total_bill']).reshape(-1,1)

y = np.array(Dataset['tip']).reshape(-1,1)

Split Train and Test

Put 1/3 to train data set and 2/3 to test data set

xTrain, xTest, yTrain, yTest = train_test_split(x, y, test_size = 1/3, random_state = 0)

Observe Test Data set

plt.title('Tips and Bills_test')

plt.xlabel('total_bill_test')

plt.ylabel('tip_test')

plt.scatter(xTrain,yTrain)

plt.show()

Observe Train Data set

plt.title('Tips and Bills_train')

plt.xlabel('total_bill_train')

plt.ylabel('tip_train')

plt.scatter(xTrain,yTrain)

plt.show()

Train Dataset / Find the best linear function

lreg=LinearRegression()

lreg.fit(xTrain,yTrain) LinearRegression()

Compare Predicted Function with Trained Data

plt.scatter(xTrain, yTrain, color = 'red')

plt.plot(xTrain, lreg.predict(xTrain), color = 'blue')

plt.title('Train vs Predicted')

plt.xlabel('total_bill')

plt.ylabel('tip')

plt.show()

Compare Predicted Function with Trained Data

plt.scatter(xTest, yTest, color = 'red')

plt.plot(xTest, lreg.predict(xTest), color = 'blue')

plt.title('Test vs Predicted')

plt.xlabel('total_bill')

plt.ylabel('tip')

plt.show()
One step forward, two steps back: Regression Testing
By: Maximiliano Merced Rios
Tags: Programming, Testing, Regression

Photo by Scott Graham on Unsplash

Testing code can be a tedious, complex, and expensive task depending on the point of execution in the life cycle of a project. Having an experienced software testing team behind the wheel can be impactful, but bugs will always fall between the cracks. How are these dormant bugs found? How can we avoid them becoming active during the next release?

What is Regression Testing?

Regression testing is the act of running functional and non-functional tests against weak points in a codebase during a new release. If one of these weak points fail a test, this action is called regression. Overall, this ensures previously developed and tested software still performs after a change. Regression testing is so impactful because not only does it test for correctness, but it also measures quality. Correctness, the result of an algorithm outputting a designated correct result, is found in all other testing methods, most notably unit and integration testing. Quality is the measurement of how new code can be implemented while old code remains functional.

Issues

One of the biggest issues of regression testing is the cost to resolve bugs at a certain point in the life cycle of the project. For example, if a core API has worked correctly for the past 5 years and a new feature reveals a bug in how it parses calls, what happens? One solution would be to adjust the calls this new feature is making so that it meets the requirements of the API. However, the API still has this bug which will eventually need to be fixed or risk security issues in the future. Another solution would be to resolve the bug in the API call parser and alert changes to all applications that depend on it. Depending on the dependencies of this API, the cost of changing call methods in other applications could be enormous and lead to new or dormant bugs (bugs that have yet to be revealed).
Secret behind Decision Tree👻
By: Sri Vigneshwar Dj
Tags: Decision Tree Regressor, Data, Regression, Machine Learning

Decision Tree is magical, do you remember Devil’s snare from first part of Harry Potter ?

Yes!!! u do right😎😎

Decision Tree has more hidden concepts than Devil’s snare, ok let’s see what is Regression first. Regression — type of ML method which can be used to predict a continuous values (in simple terms).

Why Regression Tree?

There are other models to predict continuous values like Linear, Multiple Ridge, Polynomial ,Logistic Regression and more. But when we have data points like this 👇

We can’t draw a line to predict the value because it won’t results in great predictions.

Don’t care about this Data Plotting

See here, if we use Linear regression here 👇 it’s not a good way to fit the line here.

What if we use Polynomial regression here same story. Even worst 😢

When we have more features (independent or predictors) and our data point are scattered across , we need to segregate the plot.

The segregation of data points is done using TSS — Total Sum of Squares by fitting each are every line and taking the low TSS as root.

Sample Code in python
Principal Components Regression (PRC) ใช้ PCA ในงาน Regression
By: Sasiwut Chaiyadecha
Tags: Dimensionality Reduction, Linear Regression, Unsupervised Learning, Regression, Principal Component

in All About ML
Polynomial Regression in Python
By: Sri Vigneshwar Dj
Tags: Polynomial Regression, Regression, Machine Learning

Polynomial Regression is used when we have Non-Linear Model,👇

Implementation of Polynomial Regression

import numpy as nm

import pandas as pd

import matplotlib.pyplot as plt Data_set = pd.read_csv(‘./DATA.csv’,engine=’python’)

X = Data_set.iloc[:50,0:1].values

X

y = Data_set.iloc[:50,-1].values

y

plt.scatter(X,y,color=’red’)

plt.plot(X,y,color=’green’)

Splitting the Data — Train and Test

from sklearn.model_selection import train_test_split

X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2)

Linear Regression

from sklearn.linear_model import LinearRegression

reg = LinearRegression()

reg.fit(X_train,y_train)

y_predict = reg.predict(X_test)

print(y_test,y_predict)

plt.scatter(X,y,color=’red’)

plt.plot(X_train,reg.predict(X_train),color=’green’)

plt.xlabel(‘Time in Sec’)

plt.ylabel(‘Lateral Accelerarion in m’)

from sklearn import metrics

print(“Error : “,nm.sqrt(metrics.mean_squared_error(y_test,y_predict)))

Error : 6.180035887244287e-05

print(“Intercept : “,reg.intercept_)

Intercept : 2.8366343821695255e-05

print(“Coefficent : “,reg.coef_)

Coefficent : [5.04621376e-05]

reg.predict([[1.997]])

array([0.00012914])

Polynomial Regression

from sklearn.preprocessing import PolynomialFeatures

x_poly = PolynomialFeatures(degree=3)

X_p = x_poly.fit_transform(X_train)

reg_poly = LinearRegression()

reg_poly.fit(X_p,y_train)

plt.scatter(X,y,color=’red’)

plt.plot(X,reg_poly.predict(x_poly.fit_transform(X)),color=’green’)

plt.xlabel(‘Time in Sec’)

plt.ylabel(‘Lateral Accelerarion in m’)

reg_poly.predict(x_poly.fit_transform([[1.997]]))
Multi-collinearity Key Aspect of Regression Problem
By: Akhil Anand
Tags: Multicollinearity, Regression, Machine Learning

What is Mulit-Collinearity ?

Multi Collinearity occurs when two or more independent variables(Features) are highly correlated with one another.let’s take an example.

Figure 1

Suppose you have a dataset to predict the salary of a person having independent variables Age,Year of Service.Here both the independent variable are strongly correlated and make their own relationship x1=m*x2+c.Because of mutual dependencies on one another these independent variable will be less correlated with y.

Multi collinearity won’t affect model performance but the effect of independent variables(which is multi collinear) on output variable will be less hence resulted into reduction in interpretability.

what are the cause of Multi-Collinearity ?

i. Dummy Trap

Suppose we have a dataset consists Single and Married column there is no need to take Single as an independent variable if we have taken Married as an independent variable hence resulted into redundency in dataset cause Multi-Collinearity.

It is important to eliminate first column obtained as output from one-hot-encoding otherwise it will be resulted into dummy trap.

ii. Identical Variable in dataset

Suppose we have taken two different salary columns in a dataset one represents salary in rupees other represents same salary in dollar .

iii. Creating a variable which depens on other variables present in dataset

Suppose you already have year of birth column and you have added another column onamed Age which shows high correlation on one another and reduce their indivisual effect on dependent variable.

Variance Inflation Factor

Variance inflation factor checks how one independent variable is correlated with another independent variable.Mathematically it is defined as;

R² is square of residual.

Code for V.I.F

I have taken Admission Prediction dataset to check V.I.F .

from sklearn.preprocessing import StandardScaler

scaler=StandardScaler()

from statsmodels.stats.outliers_influence import

variance_inflation_factor

scaler=StandardScaler()

def cal_vif(x):

x_scaled=scaler.fit_transform(df)

vif_data=pd.DataFrame()

vif_data["Features"]=df.columns

vif_data["VIF"]=[variance_inflation_factor(x_scaled,i) for i in

range(x_scaled.shape[1])]

return(vif_data)

cal_vif(df)

Output

we will set a threshold value. If any feature would have value greater than threshold is considered as Multi-collinear.

We can remove Multi-Collinearity between features by two process;

i. Dropping features having high V.I.F

In the above output we have two features which has high V.I.F value so we can reduce it by removing any one out of those two features.

df.drop("Chance of Admit",axis=1,inplace=True)

df.head()

Output obtained after removal of Chance of Admit

ii. Combine both the columns into one and then remove both of them

df["CGPA & Chance of Admit"]=df.apply(lambda df:df["CGPA"]-

df["Chance of Admit"] ,axis=1)

df.drop(["CGPA","Chance of Admit"],axis=1)

df.head()
การถดถอยพหุคูณ. การถดถอยเชิงเส้นแบบพหุคูณช่วยให้เราสามาร…
By: Nuthdanai Wangpratham
Tags: Regression Analysis, หลายตัวแปร, การถดถอย, Regression

I am learner and mutipotential life can contact me at nutdnuy@gmail.com
Big Data Machine Learning with SparkML — Part 2 — Ensemble Methods
By: Sailaja Karra
Tags: Spark, Python, Regression, Machine Learning

In my previous blog I showed how to setup Apache Spark in Google Colab and use Spark ML to run distributed machine learning. In this blog I am going to discuss about how to improve the efficiency of spark machine learning models using ensemble methods.

Before we jump in here is a quick take on what Ensemble methods do.

We basically train various Trees and then use these trees together to get to the final result. The below diagram shows exactly how this is done.

In SparkML we have two main algorithms for ensemble methods, Gradient Boosted Trees and Random Forests. Even though both use tree based models we will see later on that using Random Forest model not only speeds up (as the trees are trained in parallel vs GBT training one tree at a time) it also improves the model accuracy as this is less prone to overfitting.

Using the same Boston housing data set we used last week we will run both of these models.

This is what we had last week in our training data.

train.show()

We can now just take the last ‘features’ column that we built using VectorAssembler and use that. But we cannot use this directly and would have to convert this into ‘LabeledPoint’s. We do that using the following code.

from pyspark.mllib.regression import LabeledPoint rdd = train.rdd.map(lambda row: LabeledPoint(row['MEDV'] \

, row['features'].toArray())) rdd_test = test.rdd.map(lambda row: row['features'].toArray())

Here we first look at the GradientBoostedTrees.

First we import the required libraries

from pyspark.mllib.tree import GradientBoostedTrees, GradientBoostedTreesModel

Second we fit this model to our train data

gbt_model = GradientBoostedTrees.trainRegressor

(rdd,categoricalFeaturesInfo={}, numIterations=100)

We run our predictions

gbt_predictions = gbt_model.predict(rdd_test)

gbt_labelsAndPredictions =

test.rdd.map(lambda row: row.MEDV).zip(gbt_predictions)

Finally we calculate RMSE for these predictions

gbt_testMSE =

gbt_labelsAndPredictions.

map(lambda row: (row[0] - row[1]) * (row[0] - row[1])).sum()

/float(test.rdd.count()) print('Test Root Mean Squared Error = ' +

str(math.sqrt(gbt_testMSE)))

Now lets try the same thing with ‘RandomForest’ model.

model_RamdomForest = RandomForest.trainRegressor(

rdd, categoricalFeaturesInfo={},

numTrees=30, featureSubsetStrategy="auto",

impurity='variance', maxDepth=4, maxBins=32)

Once the model is trained on the training rdd we can use this for predictions.

predictions = model_RamdomForest.predict(rdd_test)

Here are the final metrics

rf_labelsAndPredictions = test.rdd.map(

lambda row: row.MEDV).zip(predictions) rf_testMSE = rf_labelsAndPredictions.map(

lambda row: (row[0] - row[1]) * (row[0] - row[1])).sum() /float(test.rdd.count()) print('Test Root Mean Squared Error = ' + str(math.sqrt(rf_testMSE)))

As you can see this gives us a much better RMSE compared to the GBT model even though we trained our RandomForest models with only 30 trees.

Hope you enjoyed reading about ensemble methods in Spark.

Happy reading !!!

References
การวิเคราะห์การถดถอยเชิงเส้นแบบง่าย
By: Nuthdanai Wangpratham
Tags: Regression Analysis, Regression, การวิเคราะห์การถดถอยเชิง

I am learner and mutipotential life can contact me at nutdnuy@gmail.com
Yelp Reviews Analysis for Bubble Tea Shops
Tags: Yelp, Naturallanguageprocessing, Regression, Data Science, Bubble Tea

Most of the bubble tea shops are located in the major cities of each state. Especially there are a massive amount of bubble tea shops in Los Angeles and San Francisco.

— Population

If we compare the number of bubble tea shops with each state’s population, we can see the positive correlation between the number of bubble tea shops and the population.

Given that bubble tea is so prevalent in the East Asian countries, we should also consider the east-Asian population in each state.

Using the U.S. census bureau’s demography data, I plot the number of bubble tea shop v.s. east-Asian population. As expected, we get a beautiful linear relationship!

— Age Group

In addition to the population, we can ask if the bubble tea shop is related to a particular age group.

Interestingly, the number of bubble tea shops has a positive correlation with the percentage aged 25–44 but seems to be uncorrelated with the Aged 14–24 group since I would naively expect that most of the customers are teenagers and young adults.

To summarize this section, I present the correlation matrix between the number of bubble tea shops, the percentage of different age groups, and the population.

Modeling

We see that several features are seemingly the crucial factors in determining the number of bubble tea shops in a city since these features positively correlate with the number of bubble tea shops. However, I want to remind readers that it’s dangerous to conclude by merely comparing the correlation! Because correlation does not always imply causality! Especially when we use two features that are causally related, i.e., population and east-Asian population. (For more details, see Spurious relationship)

To understand each feature’s importance and predict the number of bubble tea shops, I build a model through support vector regression (SVR) and use the grid search algorithm for tuning the hyper-parameters.

The dataset is split into two parts: a training set and a test set. I train the model using the training set and use the test datasets to evaluate the model.

The comparison between the number of bubble tea shops in the test dataset and the model’s prediction is shown in the left figure.

Overall, it seems that we can make a reasonable prediction by using the trained model.

To gain further insight from the model, I compute the permutation importance for each feature. The permutation importance quantifies how important a feature is by measuring how the fitting score decreases when a feature is not available. We see that the East-Asian population, the total population of a state, and the percentage of aged adults 25–44 give significant contributions.

The model for predicting the number of bubble tea shops based on demography data is a useful tool. Especially when we want to know whether the bubble tea market in a place is an oversupply or undersupply. So, don’t forget to check it before starting your own bubble tea business.
How to interpret regression variations
By: Luca Chuang
Tags: A B Testing, Experiment, Regression, Coefficient, R

5 basic interpret in the regression model

If we want to know whether the Price, Quality, and Channel affect the book Sales, we can come out with a regression like this:

Sales = a + b*Price + c*Quality + d1*Channel2 + d2*Channel3 + ε

Do channels affect sales?

1. Compare “Other Channels” with the “Reference Channel” (Channel 2 or 3 V.S. Channel 1)

Channel 2's and Channel 3’s Coefficients refer to Channel 1, therefore it only shows the coefficient of Channel 2 and 3. From their negative estimate coefficient and p-value < 0.05, we can say that:

For books with the same price and quality, Channel 2 and 3 have lower sales than Channel 1

2. Compare “Non-Reference Channels” (Channel 2 V.S. Channel 3)

From the model summary shown above, we are not able to compare Channel 2 and Channel 3, therefore, we need to run a LinearHypothesis. The null hypothesis is that the effect between channel 2 and channel 3 is the same unless we use can reject it!

After running the LinearHypothesis, the result shows that the p-value is greater than 0.05, therefore, we are not able to reject our null hypothesis.

To examine the heterogeneous price and quality effects, we have to add interactions to the regression

Sales = a + b*Price + c*Quality

+ d1*Channel2 + d2*Channel3

+ e1*Price*Channel2 + e2*Price*Channel3

+ f1*Quality*Channel2 + f2*Quality*Channel3

+ ε

Before we take a look at coefficients and p-value, let’s take a look at each scenario in different channels.

Sales(Price effect)=

b*Price (Channel 1)

(b+e1)*Price (Channel 2)

(b+e2)*Price (Channel 3)

Sales(Quality effect)=

c*Price (Channel 1)

(c+f1)*Price (Channel 2)

(c+f2)*Price (Channel 3)

Does Price effect vary across Channels?

3. Compare “Other Channels” with the “Reference Channel” (Channel 2 or 3 V.S. Channel 1)

Both of the coefficients are negative, however, only Price*Channel3’s p-value is lower than 0.05, therefore we can only say:

Price effect is larger in Channel 3 than in Channel 1

4. Compare “Non-Reference Channels” (Channel 2 V.S. Channel 3)

Same reason as before, we are not able to compare Channel 2 and Channel 3, therefore, we need to run a LinearHypothesis. The null hypothesis is that the price effect between Channel 2 and Channel 3 is the same unless we use can reject it!

P-value>0, Price effect in channel 2 and 3 are not statistically different

5. Test the significance of price effect in Channel 2

Let me bring back the Price effect scenario:

Sales(Price effect)=

b*Price (Channel 1)

(b+e1)*Price (Channel 2)

(b+e2)*Price (Channel 3)

We can see that two coefficients contribute to the result, therefore, we need to use linearHypothesis to test the significance of the coefficients. The null hypothesis is that the price effect in Channel 2 is not significant unless we use can reject it!
Python Statistics
By: Rehard Nur
Tags: Python, Regression

rehardnur/Python-Statistics

Permalink Dismiss GitHub is home to over 50 million developers working together to host and review code, manage…
Gaussian Process Regression
By: Jamie Keng
Tags: Options, Regression, Finance, Gaussian Process, Machine Learning

Now, if we plot the results in a 3D format:

The predicted call option prices and Heston call option prices are almost indistinguishable. The error term gets larger when the value of volatility approaches 0.0. It looks like it’s not an easy task to accurately predict the actual call option price under a low volatile market condition when the option is deep-in-the-money.

One observation worth mentioned is that the error term becomes stable when the call option’s time to maturity approach 0.0. My guessing is that when the time comes close to maturity, time value gradually decreases, and it is the intrinsic value that truly reflects how much the options worth. The intrinsic value of an option is a deterministic value, and therefore the error term becomes stable when the time goes near maturity.
Python ภาษาคน : Regression analysis
By: Nicky Vajropala
Tags: Python Programming, Python, Regression

Bilingual who passionately immersed in new tech.
[Study] Machine Learning — Regression
By: Doyun S Journey
Tags: Doyun Study, Regression

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore