fabiochiu/medium-articles huggingface dataset:
Minimalism is an Unattainable Dream — for a Shopaholic Like Me
By: Zo Sajjad
Tags: Minimalism, Self Improvement, Money, Life Lessons, Self

When I used to think of minimalism, my brain would always conjure up an image of a white canvas with a few specs of paint, or the newest collection of some designer selling a white tee-shirt for $100. But as I entered the decluttering phase of coronavirus induced lockdown, I began to consider what it means to adopt minimalism as a lifestyle.

Being the self-aware individual I claim to be, I know that in the past I have had a shopping problem. I love buying things. Things make me happy. for that reason, I identified as a so-called shopaholic for the past 3 years of my life. The time that quarantine gave me allowed me to dive deeper into my spending habits. Once that was done, I knew something had to change.

The idea of simple living has been around for thousands of years, but the modern minimalist movement started around 2006 with bloggers like Leo Babauta. Leo, along with others, wrote about the benefits of decluttering and promoted the idea that we could find contentment if we focused on less.

Leo wrote in an early blog post in January 2007,

According to minimalists, by removing things from our lives that don’t add value or bring joy, we can experience a lot of benefits — like better financial security, reduce stress, it can help to clarify our passions, and allow for more quality time with family and friends. But only you can decide what’s worth keeping.
Clean out your room.
By: Madeline Forster
Tags: Minimalism, Mindfulness, Cleaning, Happiness In Life, Mental Health

My story begins on a Tuesday. A Tuesday where I was sad and angry, all because of the events of the night before. I couldn’t figure out where I was going wrong. Things made me angry so easily, and it was so easy to tip me over the edge. I don’t know what happened that Tuesday morning, but everything just seemed like it was overwhelming. But then a switch flipped.

I got out of bed, changed my sheets, threw out all of the old pillows that I had. Then I decluttered to the point where my small room on its own created eleven bags of donations and waste, and just eleven bags of things that didn’t bring me joy. And I know what you’re thinking: “you had so much money, and so much waste that you just…. got rid of.” And yeah, I did. But I came to the conclusion that those things were what were overwhelming me.

Having the clutter, and the vast amount of things that were not being used, and just collecting dust. Everything that was in my room that I held in my hands, I had to justify to myself whether this was something that I needed in my life. I cleaned out my closet, and my dresser, I got rid of the things on my walls, just to make things more simple.

We spend all this time going out and buying things that we “need.” More clothes, and more items and just to use them once and have them sit there for days, weeks and even years without being touched. Clutter gives you more things to think about, more things to fill your mind, rather than the important things that we should be thinking about.

I cleaned out my room and it simplified my life. It made me realize that those things that don’t add value to your life, shouldn’t be there. Cleaning out my room on that Tuesday brought me into a happier, more open headspace, and things seem to roll off my back like they never did before. Just do me a favour, and clean out your room.
Minimalism: A Definition
By: Daniel Pohl
Tags: Minimalism, Discovery, Minimalist Lifestyle, Life, Mental Health

It’s About More Than Letting Go.

Photo Credit: Daniel Pohl

Minimalism is an age-old idea and a shiny new word. For some, minimalism is a new hipster lifestyle. A counter-culture way of living that started in the late 2000s. Their definition of minimalism is people being as comfortably close to homelessness as possible.

The truth is that minimalism has been around for generations. Henry David Thoreau (1817–1862) is one of history’s most famous minimalist. He might not have called himself one, but his simple living philosophy checked all the boxes. Socrates ( 470 BC — 399 BC), the famous Greek philosopher, was the one who said, “The secret of happiness, you see, is not found in seeking more, but in developing the capacity to enjoy less”. For these historical figures, their ideas of simple living were what would become today’s minimalism.

So what is minimalism?

Is it choosing to live as a homeless person inside a tiny house? Is it growing a beard and wandering into the woods to build a log cabin? Is it starting a blog and writing a series of books? The answer is yes.

Minimalism, as many like to define it, is a way of being more with less. I think that’s the best way to define it. It’s simple, it’s catchy, and it’s not having less, it’s being more with less. That is important to remember.

Being More with Less

Minimalism is not tossing out boxes of clutter and reducing your wardrobe to 33 items. That’s typically part of it, but it’s tossing out the reason you had boxes of clutter. It’s finding out why you needed more than 33 items in your wardrobe. It’s a path of self-discovery. It’s eliminating the habits and mindsets that filled your life with unimportant things.

How you do that is up to you. The minimalist lifestyle, as I chose to define it, is Being More with Less. Your minimalist lifestyle, however, is defined by you.

Maybe it’s selling all your earthly positions and moving into a tiny house.

Maybe it’s re-evaluating your spending habits and living more intently.

Maybe it’s deleting apps from your phone and spending more time in nature.

That’s all up to you. If you’re reading this, you’re already on the path to being a minimalist. What happens next, that’s the exciting part.

Remember, as you either start your journey or continue to grow.

Be More with Less, and above all else, Be Happy.

Thank you for reading.
You get bored before you get better.
By: Marnío A. Toussaint
Tags: Minimalism, Lifestyle, Personal Growth, Mindfulness, Purpose

You get bored before you get better.

Our world is structured differently now than it was 50 years ago. It’s fast. The flood of information that comes in on a daily basis can sometimes create a sense that were falling behind, that we have to keep up. In the process, we make movements but no progress. Ryder Carroll said “we’re working on a lot of things, but things aren’t working out.” Have you caught yourself thinking “This friend got married, that friend got a new car, my cousin went to Dubai, and in the midst of it all, I’m here with a long list of things I want to do, but can’t even seem to get done.” In my case depression, loneliness, and self hate all came from living this way. I couldn’t see the things that were right in front of me. Nothing was ever enough. I was distracted by the influx of noise calling after my attention. I’d pursue things, and when I got what I thought I wanted I was met with emptiness not too long after.

We weren’t designed to live at this pace.

When was the last time you were bored? Or physically, when was the last time you had an empty space and the option to fill it with something… anything? When is the last time you sat with a revelation and just let it sink deep? Or when is the last time you sat in complete silence… and just noticed? What’s going on with you inside, with your spouse, your family, your friends? When is the last time you just challenged a thought for weeks or months if necessary? I’ve come to realize that at large, we avoid the very thing that produces the purpose, intimacy, and creativity we are looking for — stillness. In our quest to not be bored, we sabotage ourselves. We end up in a constant cycle of reacting and we never give ourselves the opportunity to initiate.

What can we do?

We can recognize the noise. Are you anxious? Are you depressed? Are you frustrated and confused? Are you angry or afraid? Do you hide these feelings behind more hours at work? Or focusing on the kids? Or that new TV series? Even deeper, what’s the story you’ve been telling yourself?

There is a roar that comes from the time of silence we allow to happen in our lives. You gain a voice. You become confident. You find direction. You start to contribute. You discover solutions. You find out who you really are. It’s these things that are at stake when we allow the world to dictate the pace of our lives. I saw a quote that said “Time enjoyed in solitude creates the space to let in life’s wildness.” so I’ll encourage you, some of the most profound moments of your life will come from the moments you unplug.
Remove The Unnecessary
By: Karl Durrant
Tags: Minimalism, Insights, Productivity, Life Lessons, Tidying Up

Things, commitments, relationships. What promotes personal growth? Let us start there. Any farmer will agree that to grow anything well in nature, space is required. Modern western culture quickly populates the field of personal growth with its constant bombardment of knowledge, comparison, and toxic commonality. Spinning in neutral with the white noise of distraction, environmental and psychological space can elude.

Let us find space by removing the unnecessary. Drafting a list of personal, financial, health and material commitments editing can be made by asking exploratory questions such as;

What is the commitment? List a heading, e.g. Financial, material, emotional, spiritual, personal etc. If the feeling of obligation shouts, list it and explore. For a useful tool, check out www.workflowy.com.

Is this commitment imaginary? An excellent way to identify an imaginary commitment is to ask if anyone else knows of it or have I ever told anyone about it.

How many other responsibilities are connected to this? Many commitments come with a sublist of other obligations. From marriage, to own a car, the sublist also needs to be explored for excess weight.

How long has it not been acted on or used? Things and obligations gather dust. Dust it down, give it 2 months, if no value seen. Edit.

Can this be outsourced? “Busy” becomes an addiction to the point the trap of micromanagement of life becomes the norm. Who else can take some light the load? Outsourcing to family and friends is a starting point, although sometimes paying for a professional can be advantageous.

Who or what will be affected if this is not here? Holding on to commitment and obligation create a sense of purpose. Firstly, explore how “letting go” impacts personal mission for growth. After this removal of the unnecessary can be shared with others with congruence ensuring the new environmental, circumstantial and emotional space permits growth from all who are connected. Genuinely owned commitments will always be, but with a responsibility to edit the unnecessary, growth and opportunity will be fertilised with the essential space that every soul requires.
Q.O.D. Technology and Efficiency of Use
By: Allen Salazar
Tags: Minimalism, Design Thinking, Simplicity, Quote Of The Day, Technology

Love this quote by Bill Gates! It’s a reminder for us to NOT get caught up in all tools and apps that can aid in automation and overall efficiency. Rather, let us first focus on fixing and optimizing “Our Processes and Frameworks”. If we don’t identify and address the root cause; then these efforts are wasted and we are worse off then when we started.
As I downsized for my move from apartment to 188 SF Airstream I digitized everything…25 years of…
By: Stephen Chamberlin
Tags: Minimalism, Minimalist, Life Lessons, Nomad, Life

As I downsized for my move from apartment to 188 SF Airstream I digitized everything…25 years of Coast Guard awards, diplomas etc. I kept a hard copy of my social security card, trailer and truck title and birth certificate. That (aside from books which I recycle) comprises the only paper in my possession. My file cabinet is a single folder.
In my town this is how we get wood, it’s from our local nice man.
By: Moni Vazquez
Tags: Minimalism, Lifestyle, Rural, Spain, Costa Blanca

In my town this is how we get wood, it’s from our local nice man. There’s also a local bad man, but only foreigners buy wood from him–they’re Brits and northern European apparently. He is said to underpay his employees, scream at them and steal money from his customers by messing with the scale. He overcharges them. Locals know him and they warned us. It seems being a foreigner doesn’t matter if your surname is a local one; in the village they trust your ancestry, somehow. We appreciate that, it even gave us a free pan once.

We need wood again because we have no heating, a wood stove for the whole house. It felt minimalist and cool as we moved here, now it just feels cold. Yet when I look outside my window I feel peace, well worth the cold a for a few months a year.
不消費挑戰3個月心得：你的腦真的叫你一直買嗎？
Tags: Minimalism, 簡單生活, 不消費挑戰, 零廢棄, 極簡

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Info Overdose
By: Like Haffiz
Tags: Minimalism, Self Improvement, Short Read, Self Care, Personal Development

Content Shock: I think this one is really self explanitory.

Echo Chambers: we are surrounded by the same thinking, perspectives and even world views.

Constant Distraction: I use my phone more than I use my toothbrush which is a bad example but I know would there be an App which would donate 1 dollar for every pick up of my phone I would be a millionär by now! And you would be too!

Exchange FOMO (fear of missing out) for JOMO (joy of missing out). Harder to do easily said, when nowadys most of the brightest minds are all working together to ensure you are hooked on advertisement online. Check out this video to dive a little deeper into this: https://www.youtube.com/watch?v=d6e1riShmak.

Solutions:

Digital Detox Days, simply a period of time where you turn your phone off and live in the present moment. One thing you can also do is deleting the Apps everytime you dont use them, but make sure to save your passworts before hand.

Another solution would be a Timer for Phone use so you are a little more mindful about where all your time went.

Sources: https://medium.com/accelerated-intelligence/while-everyone-is-distracted-by-social-media-successful-people-double-down-on-a-totally-underrated-5a86701e9a27 https://www.bradkearns.com/2019/03/29/breakthrough/
I carried all I needed.
By: Karl Durrant
Tags: Minimalism, Poetry, Photography, Homelessness, Life Lessons

I carried all I needed. Waterproofs, sleeping bag, water, food. Life was simple then. Having nothing allowed the journey within. Keeping it light, movement without constraint. Root of it all.

Within the world “more” is sold and nothing is feared. With nothing, the self is seen. With no more noise, the inner can be heard.

Enter into the world with nothing, add knowledge, find fear. Begin again, inside and out. Unload the weight, find what is needed, where you can grow. With true connection right action finds its own reward. Life is as heavy as we make it.
Without a Smartphone can you…
By: Karl Durrant
Tags: Minimalism, Wellbeing, Time Management, Dopamine Detox, Smartphone Addiction

Wake up

Walk around the house

Eat a meal

Go to the toilet

Walk down the street

Drive a car

Focus on work

Focus on play

Connect with others

Connect with the self

Go to sleep

Some or all of these identify a dopamine addiction through validation.

Without a smartphone, gaps in time are discovered.

Question is “What can I fill it with?”

Time to “Recharge” the self for a change.
Random statement
Tags: Minimalism, Artist, Artwork, Installation, Painting

Title：Random statement

Artist: Albeswood

Project: Random statement

Creating Date: 2020

Medium: Mixed Media、

Materials: Light box、Video、Screen、Plastic material、

Size: 59×33×1.57in、150×84×4cm

Quantity: Unique（One of a kind piece, created by the artist.）

Signature: Hand-signed by artist, Signed on verso.

Photographer: Albeswood

Independent artist Albeswood believes that the human body are the most important part of daily life , Because I feel my own body , He has thought about “ why is I myself “ ，This series of works is based on such thinking . Even if the starting point is philosophical thinking , the artistic creation itself removes this kind of thinking and returns to everyday things , Let the audience feel the Zen in the quiet watching .

You can keep track of the artist’s work on instagram , And check out all the other works on his website .

Artist Website: https://albeswood.com

Instagram: https://www.instagram.com/albeswoodart

Email: Albeswood@gmail.com

All images © Albeswood
Python Kütüphaneleri
By: Merve Tatlıdil
Tags: Python, Pythoncat, Scikit Learn, Machinelearning, Pythonlibraries

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Data Normalization in Python
By: Jose Dominguez
Tags: Python, Data Science, Machine Learning, Statistics, Scikit Learn

Photo by Goran Ivos on Unsplash

When working on machine learning projects, you need to properly prepare the data before feeding it into a model. One method to perform on a dataset is normalization.

What is Normalized Data?

Normalized data within statistics in most cases involves eliminating units of measurement from a dataset. As a result, this enables you to easily compare data with different scales and are measured from different sources.

Why is Normalized Data Important?

When training a machine learning model, we aim to bring the data to a common scale and so the various features are less sensitive to each other. In this case, we can utilize data normalization as a method of transforming our data, which may be of different units or scales. This allows our model to train using features that could lead to more accurate predictions.

Methods of Normalizing Data

Simple Feature Scaling
Optical Recognition of Handwritten Digits using Scikit-Learn
By: Ayush Kumar
Tags: Machine Learning, Scikit Learn, K Nearest Neighbours, Digits Recognition

Recognizing handwritten text is a problem that can be traced back to the first automatic machines that needed to recognize individual characters in handwritten documents.

Think about, for example, the ZIP codes on letters at the post office and the automation needed to recognize these five digits. Perfect recognition of these codes is necessary in order to sort mail automatically and efficiently. Included among the other applications that may come to mind is OCR (Optical Character Recognition) software. OCR software must read handwritten text, or pages of printed books, for general electronic documents in which each character is well defined.

But the problem of handwriting recognition goes farther back in time, more precisely to the early 20th Century (1920s), when Emanuel Goldberg (1881–1970) began his studies regarding this issue and suggested that a statistical approach would be an optimal choice.
Build Data Transformation Pipelines using Scikit-learn
By: Hiran Hasanka
Tags: Python, Scikit Learn, Machine Learning

Photo by tian kuan on Unsplash

Cleaning up data is a crucial part of any machine learning procedure. A good machine learning practitioner should always be aware that data cleanup plays a vital role in preparing the data so that the machine learning model can understand the dataset better, ultimately providing better results as a model. There are many methods and procedures involved with cleaning up data, some of them very simple and some of them very sophisticated. Since datasets are unique, a unique set of processes should be crafted to transform each one of those datasets into something that can be used to train a model.

Here, I will be talking about creating a simple pipeline that will prepare a dataset to be used with machine learning algorithms. First, we’ll take a look at the dataset. I intend to use the New York City Airbnb Open Data which is available for free @ kaggle. You can download the CSV file from Kaggle and load it to a Jupyter Notebook using Pandas.

Let’s look at the dataset to identify what kind of features we’re dealing with here.

Columns of the dataset and types of data they contain

As you can see, there are 16 columns with different types of data. They need to be preprocessed accordingly. Also, you might have noticed that columns ‘last_review’ and ‘reviews_per_month’ have some significant amount of missing values. You may see that some attributes are numerical and others carry ‘object’ data. They could hold any type of python object. After loading the dataset and examining them, you’ll see that they are strings except ‘last_review’ which clearly contains dates. Furthermore, you might notice that there is a chance ‘room_type’ being a categorical attribute. Let’s check if it is categorical by counting the number of distinct values in it.

Counts of distinct values of room_type

This is in fact, a categorical attribute, containing 3 types of data.

Next, we’ll go on to plot histograms for numerical attributes of the dataset.
How to Build Custom Transformers in Scikit-Learn
By: Jake Miller Brooks
Tags: Python, Data Science, Feature Engineering, Machine Learning, Scikit Learn

FunctionTransformer

Let’s start simple with a great tool for on the fly transformations: FunctionTransformer. FunctionTransformer can be used for everything from applying a predefined function to a feature, to selecting specific columns in your feature set. The basic idea is that FunctionTransformer accepts a function (you can also pass an inverse function), and applies the function to the data via a fit_transform method. This makes it a great tool for uncomplicated transformations that can be encapsulated in a simple function; you can almost think of this as the “lambda function” of scikit-learn preprocessing. We demonstrate a few use cases below.

Selecting Features:

Here we use FunctionTransformer to select two of the thirteen features in the full Boston Housing dataset.

Output of the above Code (Image by Author)

Simple Transformations:

We were able to select the features we wanted, but perhaps we’d like to scale the values of the ‘CRIM’ (Crime Rate) feature. We could use StandardScaler, but instead we’ll apply a log scale to demonstrate how to use FunctionTransformer to apply simple functions on the fly.
The Art of Feature Engineering
By: Young Park
Tags: Polynomialfeatures, Python, Data Science, Statsmodels, Scikit Learn

One of the easiest ways to experiment with feature engineering is by using PolynomialFeatures available through Sci-Kit preprocessing module.

There are two important parameters to be aware of with PolynomialFeatures.

degree = you have the option of setting to which n degree of features you want to engineer. For instance, setting the degree = 2 means that the PolynomialFeatures will multiply each features by itself and against every other features. I personally find that interactive terms to the 2nd degree is sufficient when running linear regression but you can certainly explore by putting in a pipeline so you can GridSearch to find the most optimal n hyperparameter value. include_bias = You want to set this to False when using regular LinearRegression. Setting this to True essentially includes the y-intercept column which you do not want to do when running a regular LinearRegression out of sci-kit’s linear_model module.

One of the downsides of using PolynomialFeatures is the fast expansion of columns if you have a lot of features to begin with. One of the measures that you can take to reduce the number of features is by measuring the R2 score against the target variable. This way, you can quickly eliminate features that do not have strong predictive power.

Another method that I like to visit is to run the OLS summary using statsmodels.api module, which is another way you can run OLS in Python.

With OLS, unlike sklearn’s LinearRegression, you just have to remember to manually add in the y-intercept. After you fit the data, you can run the summary() method which produces the summary below.

P>|t| column provides the p-value of each features which can also can be used to determine the statistical significance of each features. So if you have features with p-value > 0.05, you can eliminate those from your feature selection and those with p-value < 0.05, you can keep them as part of features for your model.

Feature engineering is an exciting way to enhance your model performance and definitely requires some good discernment and experience to become more intuitively better at this. In many ways, it truly is an art. It requires some level of creativity, willingness to experiment, and explore features from many different angles.

As we all strive towards becoming better data scientists, I hope we all continue to develop and hone our skills in feature engineering.
RECOGNIZING HANDWRITTEN DIGITS USING SCIKIT-LEARN
By: Swetha Sreenivasan
Tags: Pandas, Python, Scikit Learn, Machine Learning

Here we are going to see about how the scikit-learn is used in recognizing handwritten digits.

Scikit-learn is a free machine learning library for Python. It features various algorithms like support vector machine, random forests, and k-neighbors, and it also supports Python numerical and scientific libraries like NumPy and SciPy.

four-step modeling pattern in scikit-learn:

1. Import the model

2. Create instance

3. Training the model

4. Predicting the new label data

Before modeling the data prepare and split the data for train and test. I imported the inbuilt dataset from the sklearn.

IMPORT THE REQUIRED LIBRARIES AND THE DATASET

CREATE AN INSTANCE FOR THE DATASET:

TO FIND HOW MANY ROWS AND COLUMNS USE SHAPE() METHOD:

VISUALIZING:

TRAIN SPLIT AND TEST SPLIT:

Here I split 80% of the data for train and 20% for test. Once we train our model it will recognize our new data.

USE MACHINE LEARNING ALGORITHM TO RECOGNIZE THE IMAGE:

You can use algorithm as per your choice. I’m using the support vector classifier. This algorithm comes under the supervised classification problem.

CREATING INSTANCE FOR THE MODEL

TRAINING THE INSTANCE MODEL:

PREDICTING THE NEW DATA

Once after the model data is trained use the predict method to predict the new data.

Check whether the train data and the prediction are same.

ACCURACY

To find the performance of the model import required method from the Metrics library. Metrics library has a lot of methods to calculate the accuracy such as Precision,Recall,F1-score etc.,

Here I’m going to use the Accuracy score method to calculate the performance of the model.

REPORT

Here I got 95 percent accuracy which means the model predicted 95% correctly. You can use the metrics as per your choice.

CONCLUSION

From this blog we learned how to use the sci-kit learn to recognize the hand written digits. Hope it is useful for you all. We may discuss about some other packages in another blog. If you find any mistake, please let me know. Thank you. Have a nice day.
Recognizing Handwritten Digits with scikit-learn
By: Aadesh Baral
Tags: Random Forest, Machine Learning, Scikit Learn, Digits Recognition

Recognizing handwritten text is a problem that can be traced back to the first automatic machines that needed to recognize individual characters in handwritten documents. Think about, for example, the ZIP codes on letters at the post office and the automation needed to recognize these five digits. Perfect recognition of these codes is necessary in order to sort mail automatically and efficiently.

The development of CNN has taken recognizing things to another level. But in this article, we are going to use a simple RandomForest Classifier to recognize handwritten digits.

Let's get started importing the necessary libraries and datasets.

Let's see how our digits look like using matplotlib.imshow()

Now it's time to train our model and evaluate it. As previously said we are going to use a simple Random Forest Classifier model from scikit-learn library. We will use data from 1 to 1000 to train and from 1000 to 1796 for evaluation.

After training and obtaining predicted results, let's compare results with the original values.

Our model did pretty well. It predicted 94% of the digits rightly which is quite impressive.

I am thankful to mentors at https://internship.suvenconsultants.com for providing awesome problem statements and giving many of us a Coding Internship Experience. Thank you www.suvenconsultants.com.
Give some situations where you will use an SVM over a RandomForest Machine Learning algorithm and vice-versa.
By: Harshvardhan Singh
Tags: Data Science, Machine Learning, Random Forest, Support Vector Machine, Scikit Learn

SVM and Random Forest are both used in classification problems.

a) If you are sure that your data is outlier free and clean then go for SVM. It

is the opposite — if your data might contain outliers then Random forest would

be the best choice

b) Generally, SVM consumes more computational power than Random

Forest, so if you are constrained with memory go for Random Forest machine

learning algorithm.

c) Random Forest gives you a very good idea of variable importance in your

data, so if you want to have variable importance then choose Random Forest

machine learning algorithm.

d) Random Forest machine learning algorithms are preferred for multiclass

problems.

e)

SVM is preferred in multi-dimensional problem set — like text classification

but as a good data scientist, you should experiment with both of them and testfor accuracy or rather you can use ensemble of many Machine

Learning techniques.
從圖表秒懂機器學習模型的原理：以 matplotlib 視覺化 scikit-learn 的分類器（KNN、邏輯斯迴歸、SVM、決策樹、隨機森林）
By: Alan Wang
Tags: Python, Machine Learning, Matplotlib, Scikit Learn, 機器學習

多特徵的邏輯斯函數

現在我們要回到 2 個特徵的問題；用 2 個特徵訓練完模型時，model.coef_[0] 得到的係數就會是 2 個，model.intercept_[0] 則仍是一個。你可以拿它們各別求出 2 條邏輯斯函數，不過這麼一來其實跟前面做的事是一樣的。

反而，我們稍微改一下算法，使得可以用 2 個特徵來直接算出標籤 1 的機率：

from sklearn.datasets import load_breast_cancer

from sklearn.decomposition import PCA

from sklearn.preprocessing import StandardScaler

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LogisticRegression

import matplotlib.pyplot as plt

import numpy as np

from scipy.special import expit dx = load_breast_cancer().data

dy = load_breast_cancer().target

dx = PCA(n_components=2).fit_transform(dx)

dx = StandardScaler().fit_transform(dx) dx_train, dx_test, dy_train, dy_test = \

train_test_split(dx, dy, test_size=0.2, random_state=0) model = LogisticRegression()

model.fit(dx_train, dy_train)

predict = model.predict(dx_test)

pred_prob = model.predict_proba(dx_test)

test_score = model.score(dx_test, dy_test) * 100 plt.figure(figsize=(8, 8))

plt.rcParams['font.size'] = 14

plt.title(f'Logistic regression (accuracy={test_score:.1f}%)') # 邏輯斯函數

y_t = expit(np.dot(dx_test, model.coef_[0]) + model.intercept_[0]) plt.plot([np.amin(dx_test), np.amax(dx_test)], [0.5, 0.5],

color='coral', linewidth=2, linestyle='--') # 測試集資料與其實際/預測標籤

plt.scatter(dx_test.T[0], y_t, c=predict, cmap='tab10', s=100)

plt.scatter(dx_test.T[0], y_t, c=dy_test, cmap='Set3', s=35) plt.grid(True)

plt.xlim([np.amin(dx_test.T[0]), np.amax(dx_test.T[0])])

plt.ylim([-0.05, 1.05])

plt.tight_layout()

plt.show()

就和前面一樣，這個結果和你呼叫 model.predict()、model.predict_proba() 的結果是一模一樣的。

scikit-learn 的邏輯斯迴歸對二元分類會採用 ovr（one vs. rest，一對多）策略，對 3 個以上的標籤則會用 multinomial（即 softmax），除非你用 multi_class 參數來強制指定。我想多元預測的函數求法也是一樣的，不過這就不是這篇文要討論的主題了。

邏輯斯函數的決策邊界

那麼，邏輯斯迴歸的資料預測，對最開始的原始資料會有何影響呢？下面就來在資料當中畫出決策邊界（同樣用係數和截距求出該線的方程式）。比對一下前面的圖表，便不難看出為何有些資料會預測錯誤：
Scikit-learn Train Test Split — random_state and shuffle
Tags: Python, Data Science, Machine Learning, Sklearn, Scikit Learn

The random_state and shuffle are very confusing parameters. Here we will see what’s their purposes.

First let’s import the modules with the below codes and create x, y arrays of integers from 0 to 9.

import numpy as np

from sklearn.model_selection import train_test_split x=np.arange(10)

y=np.arange(10) print(x)

[0 1 2 3 4 5 6 7 8 9]

When random_state is None and shuffle is True, each execution will result in a different set of train and test data.

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20, random_state=None, shuffle=True) print(x_train) x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20, random_state=None, shuffle=True) print(x_train)

[6 7 3 9 4 5 2 0]

[6 2 1 7 4 9 8 3]

When random_state is fixed integer and shuffle is True, the set of train and test data will be the same for each execution.

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20, random_state=7, shuffle=True) print(x_train) x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20, random_state=7, shuffle=True) print(x_train)

[0 2 1 9 7 3 6 4]

[0 2 1 9 7 3 6 4]

When we use different random_state, the output will change.

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20, random_state=42, shuffle=True) print(x_train) x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20, random_state=7, shuffle=True) print(x_train)

[5 0 7 2 9 4 3 6]

[0 2 1 9 7 3 6 4]

As you can see the output changes when we use a different random state number but it has resulted in the same train data when we have used random_state=7 thrice.

Till now the shuffle parameter was True for all cases. But what happens when we set shuffle as False?

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20, random_state=None, shuffle=False) print(x_train) x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20, random_state=None, shuffle=False) print(x_train) x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20, random_state=7, shuffle=False) print(x_train) x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20, random_state=42, shuffle=False) print(x_train)

[0 1 2 3 4 5 6 7]

[0 1 2 3 4 5 6 7]

[0 1 2 3 4 5 6 7]

[0 1 2 3 4 5 6 7]

As you can see when shuffle is False then no matter what we set for random_state, it will linearly divide the data into train and test data.

Why random_state=42?

You may have seen multiple times that most commonly 42 is used for random_state. Is there any reason for that? And the answer is NO. Python has used 42 as random_state in their documentation for example and many people have copied from the example code snippet, some people may have used it subconsciously and some may have used it lazily. So there is no particular reason for using 42 as random_state number and you are free to use any number you want.
Principal Component Analisys (PCA) con Sklearn y Python (Análisis de Componentes Principales)
By: Rubiales Alberto
Tags: Python, Sistemas De Recomendación, Recomendaciones, Scikit Learn, Matematica

Data Scientist who loves technology and business. Always learning and sharing ideas. Working at GFK as a Data Scientist. https://www.patreon.com/arubiales
Don’t underestimate these Python dunder methods!
By: Fernando Barbosa
Tags: Python, Data Science, Scikit Learn

Suppose you are testing different supervised learning techniques for a classification problem. For simplicity, I’ll assume you are familiar with these terms .

Generally, you would have a set of raw features to be used as input variables for training the algorithm. However, important transformations such as filling out missing values, standardizing variables and so forth will almost certainly take place.

As in scikit-learn’s pipelines, you could build different transformers for, say, Logistic Regression.

Notice that, even though the steps are identical and carry the same name, it is not possible to say that they are equal with regular comparison in Python.

>>> from sklearn.preprocessing import StandardScaler, MinMaxScaler

>>> from sklearn.impute import SimpleImputer

>>> from sklearn.pipeline import Pipeline

>>>

>>> mean_transformer_1 = Pipeline(steps=[

... ('imputer', SimpleImputer(strategy='mean')),

... ('scaler', StandardScaler())])

>>>

>>> mean_transformer_2 = Pipeline(steps=[

... ('imputer', SimpleImputer(strategy='mean')),

... ('scaler', StandardScaler())])

>>>

>>> mean_transformer_1==mean_transformer_2

False

And there you go! Comparing models before and after you apply different pipelines can be quite troublesome.

One way to compare them, is to compare each step’s classes to see if they are the same:

>>> class_1 = mean_transformer_1.steps[0][1].__class__

>>> class_2 = mean_transformer_2.steps[0][1].__class__

>>> class_1 == class_2

True

Now, besides ugly and verbose, this can be quite lengthy if you have a bigger pipeline. With all honesty, hardly ever do pipelines this small go to production in real life.

CAVEAT: for the sake of attention span, I kept the comparison of classes only. However, the arguments passed such as 'mean' should also be compared if you really want to get to the bottom of comparison.

And as Raymond’s famous quote states: there has to be a better way! And there is.

Let’s make a class to compare those:

>>> class SmartCompare:

... def __init__(self, pipeline):

... self.pipeline = pipeline

...

... def __eq__(self, other):

... return [m1.__class__ for m1 in self.pipeline]==[m2.__class__ for m2 in other.pipeline]

...

>>> mean_transformer_1_c = SmartCompare(mean_transformer_1)

>>> mean_transformer_2_c = SmartCompare(mean_transformer_2)

>>> mean_transformer_1_c==mean_transformer_2_c

True

Of course, this is not sooooo much better but it makes a point!

You could also inherit the Pipeline class instead of making it an attribute to your class. Or even use other methods for comparison.

In any case, dunder methods go beyond regular comparison methods and make your code more readable and pleasant.

Thanks for you patience!!! Critique is welcome, feel free to drop a comment.
PCA using Scikit-learn Python
Tags: Moa, Pca, Scikit Learn, Kaggle Competition, Data Visualization

Principal Component Analysis is a mathematical algorithm that suppress the dimensionality-reduction technique of the data

Basically, it is performed on square symmetric matrix. It can be performed in multiple ways: including Sum of squares and cross product matrix or covariance matrix or correlation matrix. A correlation matrix is used if the individual variance differ much

Main task in PCA is to select a subset of variables from a larger set, based on which original variables have the highest correlation with principal amount.

Uses

PCA is mainly used to explain the variance-covariance structure of a set of variables through linear combinations.

To find inter relation between variables in the data.

Interpret and visualize data

As number of variables are decreasing it makes further analysis simpler

Method

It searches a linear combination of variables so that we can extract maximum variance from a variable. Once this process is complete, it removes it and search for another linear combination which gives an explanation about the maximum proportion of remaining variance which basically leads to orthogonal factor. In this method, we analyze total variance.

Dataset description

I have employed Principal component analysis on the dataset from Kaggle named “Mechanism of Actions” which basically include mutations in Guanine and cytosine of different individuals and related drugs that effect human body mechanism.

Since the sample size was really large with irrelevant features so, in order to deal with this I have used PCA(Principal Component Analysis) to minimize the number of samples and extract the relevant information from those features.

S TEP 1: Importing Libraries

Following are the libraries of Python I have used in my analysis. Pandas is used to read the file / load to data. Seaborn and matplotlib for visualization

STEP 2: Loading data

file named “train_features.csv” uploaded using pandas library

STEP 3: Finding missing values:

In order to find out if data has any missing values

STEP 4: Splitting data into train and test data

Sklearn has built-in funtion to split data into train and test data. According to CRISP-DM (Cross Industry Standard Process for Data Mining) 70% of data is used for training and 30% of data left for testing purpose.

STEP 5: Scaling Feature

Here I have done Standard Scaling. Standardization is a useful for data which has negative values or data that is highly deviated. It arranges the data in standard normal distribution i.e It transform data in such a manner that its mean is 0 and standard deviation as 1.

STEP 6: Applying PCA function

PCA function is also built-in function of sklearn library in Python

STEP 6: Visualization

Since every data is incomplete without visualization. Visualization plays an important part in data science.
Handle Unknown Categories Using OneHotEncoder
By: Priyanka Dave
Tags: Categorical Data, Scikit Learn, One Hot Encoding, Data Preprocessing, Handle Unknown

How will you deal with unknown categories which were not part of your training set?

Answer: set handle_unknown=’ignore’ in OneHotEncoder

Example:

Let’s consider below as training data set:

vw_train

Here ‘Model’ is a categorical variable which we want to encode using OneHotEncoder.

Code snippet:

enc = OneHotEncoder(handle_unknown=’ignore’,sparse=False) enc_fit = enc.fit(vw_train[[‘Model’]]) enc_fit.transform(vw_train[[‘Model’]])

Output:

vw_train_transformed

Now if in testing set you found new categories then above function will automatically handle it and encode it with all 0’s.

Ultimately it is assigning new category, let’s say ‘other’ to all the unknown categories as all of them will get same encoding.

Let’s consider below as testing data set:

vw_test

Encode testing data set.

Code snippet:

enc_fit.transform(vw_test[[‘Model’]])

Output:

vw_test_transformed

It will encode all the unknown categories in same way. That means it is introducing new category from unknown categories.

Now if we will change handle_unknown to ‘error’, then it will give an error when found unknown category.

Code snippet:

enc = OneHotEncoder(handle_unknown=’error’,sparse=False) enc_fit = enc.fit(vw_train[[‘Model’]]) enc_fit.transform(vw_test[[‘Model’]])

Output:
Free hands-on tutorials to get started in Natural Language Processing 📚
By: Parul Pandey
Tags: Artificial Intelligence, Open Source, Scikit Learn, NLP, Fastai

1. Fundamentals of NLP (Chapter 1) by dair.ai

dair.ai is a community effort to democratize artificial intelligence research, education, and technologies. All their projects are freely hosted on GitHub, and they disseminate knowledge through meetups, newsletters, blogs, and study materials. One such work in progress project is on Fundamentals of NLP, which essentially will be a series on NLP principles from scratch. Currently, only the first chapter is available, and if you are an absolute beginner, this should be a good starting point. The chapter is available both in colab and as a web version. The chapter covers the following NLP basics in lucid and easy to understand the terminology:

Image by Author

After going through the tutorial, you will have a sufficient understanding of the motivation behind some of the important NLP concepts. This will set the stage for the advanced stuff.
Data Science: Pipelines
By: Alejandro Colocho
Tags: Python, Data Science, Machine Learning, Python Automation, Scikit Learn

Photo by: Casey Horner

Today we are going to be talking about pipelines. No, you have not stumble on a misclassified oil and natural gas article. You have found one of the greatest tips and tricks in data science, pipelines. Not only are these pipelines better for then environment than oil pipelines, but they are an easy way to automate some of your data science steps and combine them into one. You pass your pipeline all the steps required in your flow, and it will be performed for you. The only requirement is that your steps have a fit or fit transform method. Let’s look at a simple example.

Example

First, we are going to import everything we need.

from sklearn.datasets import load_iris

from sklearn.pipeline import make_pipeline, Pipeline

from sklearn.model_selection import train_test_split

from sklearn.ensemble import RandomForestClassifier

from sklearn.preprocessing import MinMaxScaler

from sklearn.metrics import accuracy_score

We imported the iris dataset, two pipeline makers (more on this later), train and test splitter, random forest classifier, scaling, and accuracy score (more on this later). This is all we need to make a flow in our data science project. Let’s get our data prepped to be passed into our pipeline.

data = load_iris()

X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=.3)

We loaded the dataset and split it into a train set and test set. We did not clean the dataset because we already know is clean. All we have left is to split it into the two different sets. Now, let’s get to the fun stuff.

pipe = Pipeline(steps=[('scale',MinMaxScaler()),('RFC',RandomForestClassifier())])

pipe.fit(X_train,y_train)

preds = pipe.predict(X_test)

First, we have to make sure that our steps have a fit or fit transform method. Then, we have to add them to a list of tuples in the following format (name,initialized class). After, we use the fit method to fit our data to the MinMaxScaler and to our RandomForestClassifier. After we fit it, we can use the predict method to estimate our target variable for X_test. Let’s look at our other pipeline.

other_pipe = make_pipeline(MinMaxScaler(),RandomForestClassifier())

other_pipe.fit(X_train,y_train)

other_preds = other_pipe.predict(X_test)

So, what’s the difference? I’m glad you asked. The only difference is that in make_pipeline you do not have to pass the classes in a list or name them. To prove they do the same thing, we will benchmark both predictions based on accuracy.

print(accuracy_score(y_test,preds))

print(accuracy_score(y_test,other_preds))

The wrap up

Just like that we created a pipeline for our sample data science project. It’s incredibly easy and fast. This will allow your projects to be more organized and automated.