fabiochiu/medium-articles huggingface dataset:
Random Data Science Questions #2
Tags: Python, Data Science, Kaggle, Pandas, Numpy

It’s that time again. Welcome back to Random Data Science Questions, episode 2, where do I begin analyzing my data? I’ve tried to not get too into the code this early, but we knew that it was an inevitability. So, we will start off with something simple and commonly used.

Imagine this. You just get home after downloading a sweet new dataset from Kaggle. You eagerly open your laptop and load up your Jupyter Notebook, salivating at the mouth while the anticipation builds. But wait! Looking at a brand new dataset is almost intimidating as looking at a blank canvas. Where do we start from here?

No worries, I got you. Assuming that the data you are working with is labeled in some form or fashion I have the perfect start to your blank canvas. I call it DISH but I pronounce it like dish(-_-). Since the first step is always the most intimidating, I used DISH, which stands for describe, info, shape, head, as a method of getting the ball rolling. These commands are preceded by the dataframe, which is commonly represented in documentation as df. For me, DISH was like getting the dish on the data. Lastly, before we get into the code, none of these commands would be possible without the Pandas Library. I’m thinking that an RDQ on the Pandas Library is in order.

So, because the Titanic is so simple, we’re going to use it as a reference.

#lets load our data in

df = pd.read_csv('train.csv')

Now that we have our data loaded, let’s take a peek at what we’re dealing with using DISH.

#let's look at decribe

df.describe()

df.describe()

What’re we looking at? Well, pd.describe is a pandas function that runs over our data and gives us a quick overview of each label (columns) in the the data. We can see the count (number of items), mean(avg), standard deviation (how far apart we can expect the data to spread), and split data on minimum through maximum amounts. It is here we can begin to extract small but useful data like the average age (29.6) or the the oldest recorded person aboard (80). Now we have our first brush stroke onto the canvas.

Next is info.

df.info()

df.info()

Info gives us an overview of of the data types that we will be working with. It also gives us the amount of non-null values under each label and the amount of entries.

Then we have shape.

df.shape

(891, 12)

Shape, represented as a tuple, tells us the number of rows and columns in our data. In this particular case, we have 891 rows and 12 columns. What I like to do is save the number of rows and columns in their own variable to reference.

num_rows = df.shape[0]

num_cols = df.shape[1]

Lastly, there’s head.
A Simple Way To Get Started PyTorch — Lightning for Deep Learning
By: Shivanand M N
Tags: Python, Kaggle, Pytorch Lightning, Pytorch Tutorial, Deep Learning

A tool that helps to structurize your code PyTorch complexity.

For the introduction of PyTorch-lightning read this post from the creator.

Let's start with PyTorch-Lightning library installation

pip install pytorch-lightning

Pytorch is the best library on making Pytorch organized

No Boiler Code like — .to(device) , .step() , .backward()

, , Simply implement train, validation, testing loops

Change device from CPU to GPU or TPU by changing the single argument.

Separate modelling and data handling/processing

Every epoch checkpoints can be stored and visualized in tensorboard

Resume your training model any time from saved checkpoints.

Importing library as pl import pytorch_lightning as pl

There are many reserved methods in the lighting modules called

*hooks*:

configure_optimizers — this should return optimizer(Adam/SGD).

— this should return optimizer(Adam/SGD). training_step — training loop, takes batch and batch_idx as parameters.

— training loop, takes and as parameters. validation_step -validation loop, takes batch and batch_idx as parameters.

-validation loop, takes and as parameters. testing_step - testing loop, takes batch and batch_idx as parameters.

Simple Model

Pytorch-lightning has data module extension that structures your data preprocessing. This structure helps to read and understand code easily for everyone.

It helps to reuse data across multiple projects even with complex data transform and multiple-GPU handling

*Hooks:**

train_dataloader()

val_dataloader()

test_dataloader()

Above methods in lightning data-module are data-loaders, those methods should return DataLoader() .

prepare_data() — Download and tokenize or do preprocessing on the complete dataset, because this is called on single GPU if your using multiple GPU, data here is not shared across GPUs.

— Download and tokenize or do preprocessing on the complete dataset, because this is called on single GPU if your using multiple GPU, data here is not shared across GPUs. setup() — splitting or transformations etc. setup takes stage argument None by default or fit or test for training and testing respectively.

It simplifies to two class objects required to train model and test it on test data.

Now, we can use ` pl.Trainer ` for the instantiating trainer with many parameters like `max_epochs, tpu_cores , num_gpus , cpu (default), logger etc.

` for the instantiating trainer with many parameters like `max_epochs, , , (default), etc. trainer.fit , fitting the data on model

Pytorch-lightning automatically saves the model as checkpoints as .ckpt files. you can manually save a model as trainer.save_checkpoint(“model.ckpt”) and reload back to the model as pl_model.load_from_checkpoint(checkpoint_path=”model.ckpt”)

PyTorch model is more professionally robust even for mature developers with a lot of parameters/ hooks are available to focus on the better model rather than struck at boilerplate coding.

Here are the Pytorch-lightning implementations :
[Open] I’m the Best Coder! 2020 蝦皮數據分析競賽解題分享
By: Nancy Sw
Tags: Python, Kaggle, Machine Learning, Shopee, Kaggle Competition

It's never too late to start.
A Quicker Way to Download Kaggle Datasets in Google Colab
By: Nabanita Roy
Tags: Dataset, Kaggle, Data Science, Kaggle Api, Google Colab

Step 3: Run the script

!pip install -q kaggle

!pip install -q kaggle-cli

!mkdir -p ~/.kaggle

!cp "your/path/to/kag gle.json" ~/.kaggle/

!cat ~/.kaggle/kaggle.json

!chmod 600 ~/.kaggle/kaggle.json # For competition datasets

!kaggle competitions download -c dataset_name -p download_to_folder # For other datasets

!kaggle datasets download -d user/dataset_name -p download_to_folder

Replace:

your/path/to/kag gle.json with your path to kaggle.json on drive. You can explore your drive on the left and copy the path. I have double quotes around it to avoid issues with blank spaces in folder names.

with your path to kaggle.json on drive. You can explore your drive on the left and copy the path. I have double quotes around it to avoid issues with blank spaces in folder names. download_to_folder with the folder where you’d like to store the downloaded dataset

with the folder where you’d like to store the downloaded dataset dataset_name and/or user/dataset_name as follow:

To download a competition Dataset: You can easily get hold of the dataset_name to use in the URL. For example, if you want to download Fake News dataset, select just fake-news from the URL :

Also, make sure to have agreed to the competition rules:

Then, your final script would look like -

!pip install -q kaggle

!pip install -q kaggle-cli

!mkdir -p ~/.kaggle

!cp "/content/drive/My Drive/your_path_to_kaggle.json" ~/.kaggle/

!cat ~/.kaggle/kaggle.json

!chmod 600 ~/.kaggle/kaggle.json # For competition datasets

!kaggle competitions download -c fake-news -p Dataset

If you do not accept the competition rules then you’d encounter the 403 Forbidden Error.

To download any other Dataset: Replace user_name/dataset_name with the Kaggle username and the dataset name. This can be easily extracted from the URL. For example, if you want to download US Election 2020 Tweets, you could simply copy the part after kaggle.com -

… and plug it into the script.

Therefore your final script would look like -

!pip install -q kaggle

!pip install -q kaggle-cli

!mkdir -p ~/.kaggle

!cp "/content/drive/My Drive/your_path_to_kaggle.json" ~/.kaggle/

!cat ~/.kaggle/kaggle.json

!chmod 600 ~/.kaggle/kaggle.json # For other datasets

!kaggle datasets download -d manchunhui/us-election-2020-tweets -p Dataset

Finally, to re-run notebooks without having the necessity to scroll up, you could comment out the entire script including codes for unzipping datasets. If you are logged into your Google account, have access to your drive, you can run your code on any machine by directly downloading data without worrying about kaggle.json configs.
Kaggle: Instacart Market Basket Analysis
By: Jigar Parmar
Tags: Kaggle, Data Science, Machine Learning, Data Analysis, Data Visualization

Machine Learning Problem

The data science team of Insta-cart here wants the Machine Learning

Engineers or the Researchers to come up with new methods to suggest the

probability of the product the user is likely to “buy again”

The dataset consists of 6 files, each containing different information of the

insta-cart. It contains the information about the products, the aisle to which

the product is kept, what product was reordered, after how many days did

the user come to shop etc.

The dataset is divided into 3 parts Prior, Train, and Test.

Prior order contains the information of the previous orders from the users, and the last order from the user is either added in Train or Test set. The number of orders from the user range from 4 to 100, there are almost 50K products and 3M orders from the users

The dataset is not balanced in terms of ‘reordered’, Neither the number of

orders of a given product are the same as others. Here there is also a

chance that the customer may order nothing from his / her previously

ordered products. So ‘None’, can also be an answer to a user’s next

purchase. Thus we should also consider None as a different product along

with others.
¿Qué es Kaggle y cómo puedes entrar en esta comunidad?
By: Planeta Chatbot
Tags: AI, Kaggle, Español, Inteligencia Artificial, Podcast

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Getting To Know Your Features In Seconds With RAPIDS
By: Louise Ferbach
Tags: Kaggle, Model Building, Data Science, Machine Learning, Computer Science

Exploratory data analysis, and especially getting an idea of features importance, is a crucial step when confronted with a data science problem. However, this step can often be trying if your database is huge, with numerous interconnexions, making the analysis costly in computing time.

GPUs are great at improving computing time… if you can use them ! Indeed, their use is often dedicated to training neural networks, thanks to powerful frameworks (Tensorflow or Pytorch are the most famous) that have enabled millions of developers to unleash their incredible powers.

However, if you have some dirty preprocessing to get done with pandas or scikit-learn, that can be quite heavy if your data is big, since these packages do not feature any parameter helping you run them on GPUs. To sum things up, you won’t be able to get the most out of your GPU at this stage.

RAPIDS is a suite of packages developed by Nvidia AI, that intends to execute end-to-end data science and analytics pipelines entirely on GPUs.

The goal of this notebook is to perform univariate regressions of each target on every feature in the MoA competition, namely 872 x 206 = 179632 logistic models to estimate separately.

The good news ? This is possible within minutes with RAPIDS !

You can note that I never import neither pandas (replaced by cudf) nor scikit-learn (replaced by cuml).

This notebook will be a tutorial, aiming at helping you get familiar with the use of the libraries. At the end, you’ll be able to select, among your features, those which appear to be the most crucial in predicting your targets. That will enable you to build robust, medium-sized models with better interpretability.

If you want to skip the processing and jump to the results, you can directly take a look at the results here.
How to Become a Data Scientist
By: Yishai Rasowsky
Tags: AI, Kaggle, Data Science, Machine Learning, Artificial Intelligence

In the next five minutes, you will have the secret to becoming a data scientist.

You want to find your first job. But who will hire you without experience.

Try to produce your own project that you can showcase on Linked-In.

If you don’t have skills yet, check out tutorials from Youtube, or Coursera. I also personally recommend Kaggle and Laurence Moroney.

Once you get your first job, you will learn skills from your colleagues much faster than when working on your own.

Wishing you success!
5 Stories Data Tell us About Data Scientists
By: Éverton Bin
Tags: Python, Kaggle, Data Science, Machine Learning, Storytelling

5 Stories Data Tell us About Data Scientists

Photo by Dariusz Sankowski

It may sound like the revenge of structured data, but it’s actually just a survey conducted by Kaggle Platform. The data of the 2019 Kaggle Machine Learning and Data Science Survey was made available here and used so we could listen to the stories they could tell us about data scientists.

These were the questions that guided us through this analysis:

What is their educational background?

Which activities do they perform in their daily work?

Which tools do they use?

And what about their salary?

And here are the stories they told us:

Education

Image by Author

I know this first figure may look like bad news for you if you’re just getting started in the journey to become a data scientist. I don’t want to give you any spoiler, but if you feel disappointed now, go check the last section called Education x Salary and it may calm you down again.

Most of the data scientists who answered the survey has indeed a Master’s degree on their back. Still, it is very common that they attend online courses through popular platforms like Coursera, Udemy, DataCamp, Udacity, and many others.

Image by Author

It could indicate that, no matter what is your formal educational background, you will have to continuously seek knowledge, especially in this technological field that brings us news almost every single day.

Activities

When hearing about Data Science, it is inevitable to think about the hype of artificial intelligence and machine learning algorithms. But when it comes to a regular day in a data scientist’s life, would that be the main task?

Image by Author

Over 60% of them actually have to deal with data analysis instead of just worrying about what algorithm to use or finding a way to improve their models. Actually, the survey shows us that it doesn’t matter if you are a data scientist or a data engineer, data analysis will play a big role in your activities.

Also, we can see that almost 35% of the data scientists who participated in the survey performs activities related to data infrastructure. That shows how different roles in the Data Science field relate to one another.

Tools

Probably it would be no surprise if I would tell you that Python is the most popular programming language among data scientists. But what if I asked you what is the second language they use the most?

Image by Author

If your answer was SQL, you got it right! SQL, or Structured Query Language, is a simple but powerful language, and its commands can be grouped into four main functions: data definition language (DDL), data manipulation language (DML), data control language (DCL), and data query language (DQL).

It shows us that a great part of the work for a data scientist is to understand data, by executing queries in databases, transforming, manipulating, and analyzing them.

But language is just one of the tools that we can find in a data scientist’s toolbox. What about the most used databases and machine learning frameworks?
Binary Classification: Tips and Tricks from 10 Kaggle Competitions
By: Patrycja Jenkner
Tags: Kaggle, Data Science, Machine Learning, Binary Classification, Tips And Tricks

Imagine if you could get all the tips and tricks you need to tackle a binary classification problem on Kaggle or anywhere else. I have gone over 10 Kaggle competitions including:

– and pulled out that information for you.

Dive in.

Modeling

Dealing with imbalance problems

Metrics

Loss

BCE and Dice Based

Focal Loss Based

Custom Losses

Others

Cross-validation + proper evaluation

Post-processing

Ensembling

Averaging

Averaging over multiple seeds

Geometric mean

Average different models

Stacking

Blending

Others

Repositories and open solutions

Repos with open source solutions

Image based solutions

Tabular based solutions

Text classification based solutions

Final thoughts

Hopefully, this article gave you some background into binary classification tips and tricks, as well as, some tools and frameworks that you can use to start competing.

We’ve covered tips on:

architectures,

losses,

post-processing,

ensembling,

tools and frameworks.

If you want to go deeper, simply follow the links and see how the best binary classification models are built.
Simple Data Engineering to Improve Your Machine Learning Results
By: Kr. Wan
Tags: Kaggle, Data Science, Machine Learning, Neural Networks, Deep Learning

Disclaimer: the following content is largely thanks to this amazing notebook. Make sure to check it for implementation details!

Rank Gauss

Various, numerous, and many machine learning techniques, such as PCA, actually rely on the assumption that the underlying data is normally distributed. With a dataset that is not normally distributed, the usefulness of such techniques will be negatively affected. RankGauss, a technique developed by Michael Jahrer in Porto Srguro’s Safe Driver Prediction, is a solution to such problem. I will not dive into the math behind it (because I am not good at Math!) but the purpose of RankGauss is to transform a non-normal distribution to a normal distribution, and it usually works better than standard standardization. Let us see the effect first!

Here is the distribution before applying RankGauss:

Pre-RankGauss Distribution

Here is the distribution after applying RankGauss:

Post-RankGauss Distribution

As you can see, the feature distribution has been normalized after applying RankGauss.
House Price Prediction Machine Learning Model.
By: Syed Nauman
Tags: Kaggle, Machine Learning, House Price Prediction, Board Infinity, Médium

Thousands of houses are sold every day. There are some questions every buyer asks himself like: What is the actual price that this house deserves? Am I paying a fair price? In this project, a machine learning model is proposed to predict a house price based on data related to the house (its size, the year it was built in, etc.). During the development and evaluation of the model, I have demonstrated the code used for each step followed by its output.

Note:-Here I’m using Jupyter Notebook. You can even use Google Colab or any other ID. I tried to add comments in all the cells so that you can understand them easily. If you still have any doubt you can reach me through the links given at the end of this blog.

I have divided the blog into Simple steps. In order to make you all more clear.

Import the necessary libraries and loading data.

Data is rightly skewed hence Transforming it

VISUALIZING NULL VALUES

FEATURE ENGINEERING

filling missing values

LOADING LIBRARIES

Modeling:

1. Gradient Boosting: Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. It builds the model in a stage-wise fashion like other boosting methods do, and it generalizes them by allowing optimization of an arbitrary differentiable loss function.

2. XGBoost: XGBoost is an optimized distributed gradient boosting library designed to be highly efficient, flexible and portable. It implements machine learning algorithms under the Gradient Boosting framework.

3. LASSO: Lasso regression is a type of linear regression that uses shrinkage. Shrinkage is where data values are shrunk towards a central point, like the mean. The lasso procedure encourages simple, sparse models. This particular type of regression is well suited for models showing high levels of multicollinearity or when you want to automate certain parts of model selection, like variable selection/parameter elimination.

4. RIDGE: Ridge Regularization is a technique that helps overcoming over-fitting problem in machine learning models. It is called Regularization as it helps keeping the parameters regular or normal.

The score of the model is 0.12503

Conclusion/ Results

As a result, we can say, XGBOOST is the best fit to this dataset, gives 0.107 accuracy. Gradient, lasso, ridge Models are working very well on this Dataset. Ensembled all the models together for better results.

Future Scope

In future, the models can be upgraded with some better techniques in terms of getting higher and better accuracy.
Finishing 2nd in Kaggle’s Abstraction and Reasoning Challenge
By: Alejandro De Miquel Bleier
Tags: Artificial Intelligence, Kaggle, Artificial General Intell

The three tasks above were part of the dataset of the ‘Abstraction and Reasoning Challenge’ (ARC), a competition launched by Kaggle in February 2020. Hosted by Keras’ creator François Chollet, it aimed at dealing with some of the main gaps that currently exist in the world of AI, as exposed in his paper ‘On the Measure of Intelligence’. In particular, participants were challenged to develop a program capable of solving IQ test-like problems that it had never seen before. Humans are very good at quickly detecting all sorts of visually clear patterns and symmetries that aren’t obvious to a program if it is not taught to detect them. A program capable of achieving human-like performance in such tasks would bring us a step closer to what is known as Artificial General Intelligence (AGI).

In this post I will share my personal experience in this competition, in which I managed to finish 2nd with the help of my teammates. I will also share the main ideas of our algorithm, which can be found on this GitHub repository.

Understanding the Dataset. First Approaches

A training dataset of 800 tasks was given, and a hidden test set of 100 extra tasks would determine the competition’s final leaderboard. For each task of the test set, one was allowed to submit three possible solutions, and if any of the proposed solutions was correct, the task would be considered solved.

Quite early on, some high-quality notebooks were shared by some of the competition participants pointing at the potential usefulness of combining CNNs and recurrence. This notebook in particular was my first source of inspiration for trying some DL-related approach. However, after a few attempts, it became clear to me that it was quite limited. Whereas some of the tasks could clearly be solved with such an approach, some others were quite unlikely to be solved by a generic CNN. In particular, I didn’t manage to solve a single task of the test set with this approach.
Easily Import Kaggle Datasets in Google Colab with Python
By: Joseph Lilleberg
Tags: Kaggle, Machine Learning, Google Colab, Towards Data Science, Deep Learning

Photo by Filiberto Santillán on Unsplash

When I first tried to import Kaggle datasets into Google Colab, I scoured Medium, Kaggle, and other online forums for how to do so and each answer was slightly different from the next. I also found it rather tedious to try to remember all the steps in order to successfully import the dataset.

Therefore in this post, I will share my solution to these problems with a function I wrote to handle all the dirty work for you. All you have to do is provide the Kaggle dataset url and your api credentials. If you would like to understand the script, please read the entire post. Otherwise, the script in it’s entirety can be found at the end.
[Kaggle] Predicting Housing Price — a TensorFlow approach (2nd try)
Tags: Kaggle, Machine Learning, TensorFlow

Continuing from my previous post, after selecting training features, I continued with TensorFlow.

Import the necessary modules and subset the data.

Then I did a train test split with frac=0.8, with SalePrice as the label and else as features.

I then used normalization to transform the feature data, so that they are on the same scale. (It is faster and easier for gradient descent to converge.)

Then I defined a function of building and compiling a neural network with 2 hidden layers with ReLU (Rectifier Linear Unit) activation function, which is actually an advanced version of sigmoid function. Then stack the normalized data into the model.

Fit the training data. and plot the loss over time.

Here are some visualizations of the true value and the predicted value:

Then repeat the same for the test data for uploading.

https://github.com/schneeboat
Meteorological Data Analysis
By: Kumar Vaibhav
Tags: Kaggle, Data Science, Meteorology, Data Analysis, Data Visualization

In this we used a meteorological dataset to analyze the data of 10 years and made a conclusion on it. Dataset which is used in this, obtain from Kaggle (https://www.kaggle.com/muthuj7/weather-dataset). Here we have used the NumPy, Pandas, Matplotlib and Seaborn to load the dataset and to plot the graph.

In this we have recorded and plotted the graph between Apparent Temperature and Humidity of different months.

At first we have loaded my python libraries, which will be going to use further.

Importing the libraries of Python

Here, our dataset looks like this after loading with help of Pandas.

First five data from dataset.

Now we will check the shape of the dataset and information’s about the datasets.

It gives the shape structure and basic info about data.

Then we will check the any null value is present or not, if it is present and it is relevant to our analysis then I have to manage and enter some value on it.

Checking presence of null value.

After checking the null value, I will remove the unwanted data to make our analysis more easy. Here ‘axis=1’ means removing of column listed under drop function.

In dataset, a date column was given. So firstly we will make a proper format so that python can treat those date to a proper functional date.

Conversion of date into a proper format.

Now we will plot our first graph.

As we can see there are no such any changes in temperature in 10 years, and this graph tell that in 2012 the minimum temperature was -8 degree Celsius.

Graph between year and temperature

Now we will plot the graph in month of February 2012, we have seen that temperature declined sharply up to -10 degree Celsius.

Graph between temperature and year in month of February.

In below graph, we can see that there is there is not much changes occur in last 10 years in the month of August. Temperature minimum rise and fall can be seen here.

Graph of August

Conclusion: While looking all these data and graph we conclude that there are not major changes has been seen in between 2006 and 2016 due to global warming. Humidity remain constant throughout the years with minute amount of rise and fall.
Let’s take a look of Seattle AirBnB Open Data using Python
Tags: Python, Kaggle, Airbnb, Data Analysis, Data Visualization

Summary

Seattle AirBnB Open Data describes the listing activity of AirBnB homestays in Seattle, WA until 2016. It is consists of 3 files

listings.csv — including full descriptions and average review score

calendar.csv — including listing id and the price and availability for that day

reviews.csv — including unique id for each reviewer and detailed comments

How do seasonal costs change?

Below graphs show the mean, quantile prices by months.

Average prices are highest in summer and increase slightly in December. Perhaps summer holidays, year-end events, and Christmas were influential. Prices fluctuate mainly at high price points, and cheap rooms do not change even when the season changes.

Which features are most related to homestay cost?

By using `pandas.DataFrame.corr` can easily do correlation analysis. Let’s show the heatmap of correlation matrix.

Correlations related to reviews

Each score of review features has strong correlation.

Number of reviews and price has a little negative correlation. Expensive homestay are expected to no attract many people and have fewer reviews.

Correlations related to bathrooms, accommodates, bedrooms, beds

The number of bedrooms, the number of bathrooms, and the number of beds have a high correlation with each other and a high correlation with price. It’s natural to think that the more rooms, beds, and toilets you have, the more expensive it becomes. However, the number of people was the most influential among them (0.69).

(0.69). Whether or not a homestay can be rented for a long time has no significant correlation with other features other than each other. The minimum and maximum stay dates are not significantly correlated with other features.

What is different between superhost and regular host?

Superhosts almost slightly outperformed all score of reviews. However, there are far more reviews written. It could be because more customers are staying at accommodation on the superhost, or it could be because the superhost encourages customers to actively write reviews. Another difference is that on average the superhost has fewer rooms than the rest of the hosts. Even though the average number of beds, toilets, and number of people in a room operated by superhosts are slightly smaller There is a shorter minimum/maximum period for customers to stay, and more guests can be included. But the difference is not big.

Assuming that the higher the number of reviews, the more customers it attracts, the more likely a superhost will have other factors that can attract more customers. For example, they may have been running a homestay for a long time, or their location may be better.

Date the host started hosting
How my first internship helped me more than I ever thought
By: Charles Samuel R
Tags: Kaggle, Data Science, Internships

Internships. A phase of learning in every person’s life where we can prove our skills as a person and I got a chance to show exactly that and boy was it a good one.

Let me set the scene for you:

July 2020:

My college St. Joseph’s College of Engineering announced an internship happening at this company called TactLabs led by the amazing Raja CSP Raman for 3 months from July to October.

First, there was this introductory session where they talked about what they do at the company and I completely missed it so I can’t talk about that all too much (Quarantine messed me up badly 😅). Next, we had to send in our resume to one of our placement coordinators to set up the interview. I was one of a bunch of students who were called up for the interview.

The interview:

It was fairly straightforward. We had to talk about yourself and show a few projects that we had done if any. I was then selected for the second round where the fun was.

Round 2:

This round was all practical as we had to develop a web app that would convert and image to text. It involved using Flask (A subject I just started learning). I managed to complete the base of the app but didn’t know much about how to Dockerise it. (I didn’t have the best OS for that in my defense)

The result:

We were all called back again one by one to know if we were selected or not. I was happy to announce: I GOT THE JOB!!!! 🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉

Let’s get down to what I did during the entire internship to date.

The Internship (July-October)😃

It was announced that we were split into teams like the Front-end team, ML/DL teams, and so on. I was assigned to the Front-end and ML/DL team. My involvement in both these waned as the internship went on for a VERY good reason. My friend Kamal decided to ask for work related to NLP and that lead to me and him getting NLP work and we never looked back after that.

There are 4 things I would like to say about my time here at TactLabs:
Performing Analysis of Meteorological Data
Tags: Sentiment Analysis, Kaggle, Data Science, Data Visualisation, Python3

In this blog we are going to analyse the data from the Weather data-set of Finland, a country in the Northern Europe. You can find the data-set on Kaggle (https://www.kaggle.com/muthuj7/weather-dataset).We are going to use the numpy, pandas and the matplotlib libraries of Python.

Following is the Hypothesis of the Analysis : “Has the Apparent temperature and humidity compared monthly across 10 years of the data indicate an increase due to Global warming.”

Let us start by importing the required libraries and our data-set:

Libraries required for analysis

Importing our data-set

Here is a small preview of how our data-set looks:

First 5 entries of our data-set

Now we need to drop the unwanted data, convert the data in to our need and resample our data :

Here is how the data looks after resampling:

First 5 entries of resampled data-set

Now let us plot our data in a line graph

As we can see, both the peaks and the troughs are almost same throughout the period of 10 years. Here is a plot of the average temperature and humidity of the month of April over 10 years.

We can clearly see that there is a sharp rise in temperature in the year of 2009 whereas there is a fall in temperature in the year of 2015. Hence we can conclude that global warming has caused an uncertainty in temperature over the past 10 years while the average humidity as remained constant throughout the 10 years.

I am thankful to mentors at https://internship.suvenconsultants.com for providing awesome problem statements and giving many of us a Coding Internship Exprience. Thank you www.suvenconsultants.com.
My Journey from Novice to Kaggle Master 🤓 within 1 month 💪
Tags: Kaggle, Data Science, Machine Learning, Featurepreneur, Notebook

One-day Hackathon 📅

Few days before the hackathon, I created an account in Kaggle and Tactlabs guided me to level up from Novice to Contributor. It is very easy and it will take just 10 to 15 minutes.

During the hackathon, we were given a set of datasets and the task is to show our Machine learning skills in those datasets. We have to analyze those datasets, visualize those data, and to use ML algorithms to predict something from those data. Intern members were grouped into teams and each team has 2 to 3 members. It started at 9 am IST and last long till 7 pm IST. We worked that whole day with sufficient rest taken at intervals.

We have to work in Kaggle notebooks and we have to publish them at the end. We enjoyed the whole hackathon and also learned a lot. Though we ended as a Runner up, we got satisfied by gaining a wonderful experience and also learned many new things.
Solution to Kaggle’s Ames House Price Competition:
Tags: Kaggle, Data Science, Machine Learning, Artificial Intelligence, Predictive Modeling

An Exploration of Label Encoder

I have been experimenting with the Kaggle House Price Competition for several months now and I have seen several different methods that have been approached to predict on the house prices in the dataset provided for this competition. The website address to the competition and accompanying datasets is:- https://www.kaggle.com/c/house-prices-advanced-regression-techniques/overview

One solution I found to predict on the sales price was to make the the living space area the only independent variable. Another answer I found was to separate the categorical values from the numerical values and to only predict on the basis that only the numerical values were made the independent variables by which to make a prediction on.

After much reflection and study, I endeavoured to develop an alternative solution that yielded an accuracy of 94.22% when tested on the validation set. This method I explored was to use the LabelEncoder.

LabelEncoder is a preprocessing module within the Python sklearn library, which was designed to be used to encode target variables, such as the y variable. One use for LabelEncoder is to transform non-numerical values into numerical labels, which is what I have done in this exercise.

Sklearn provides an example of how to use LabelEncoder to convert objects to numerical features:-

The train file of this competition question has 81 columns, with 43 of those columns being test based. I therefore was required to use the LabelEncode function on the object columns on both the train file and test file, making a total of 86 columns:-

In addition to encoding the text based columns, I also incorporated feature selection in an attempt to improve my score, dropping a few column that had null values in them:-

After preprocessing the data and selecting a few features, I defined my X, y, and test variables:-

Once the dependant and independent variables had been defined, split the test set for training and testing:-

Once the train dataset had been split for training and testing, I used the XGBRegressor function to endeavour to achieve optimum accuracy:-

Upon testing on the validation set, I obtained an accuracy of 94.22%:-

Of course, more work needs to be done to improve on this score, but it is certainly a good start. The full code to the Jupyter notebook this program was written in has been made public and can be found on the Kaggle website:-

https://www.kaggle.com/tracyporter/ames-house-prices-xgb-regressor
How to Load data on Kaggle Notebook?
By: Yanwei Liu
Tags: Kaggle, Data Science, Machine Learning, Artificial Intelligence, Deep Learning

Before showcasing your awesome data science project to Kaggle, please make sure that the dataset is successfully loaded on Kaggle Notebook.

You have two options:

The first one would be selecting existed dataset or uploading your own dataset. ( File-> Add or upload data)

and then use the following code to figure where your dataset is

import os

for dirname, _, filenames in os.walk('/kaggle/input'):

for filename in filenames:

print(os.path.join(dirname, filename))

finally
Sleeping Sound in Seattle: How to Become a Top Earner on Airbnb?
By: John Abel
Tags: Python, Kaggle, Data Science

Does it help to be an Airbnb Superhost?

You might be thinking — what exactly is a Superhost, and how do you become one? A Superhost is essentially an experienced host on Airbnb, where they provide extraordinary experiences for their guests and provide a shining example to other hosts.

How important is this status to potential guests?

Looking at the figures above, it appears that Superhost status has little effect on the listing price of the property. Albeit, Superhosts had a slightly higher average listing price compared to those hosts with non Superhost status.

However, when looking at the future availability of the listings differences between Superhost and non-Superhost hosts begin to appear. Where non-Superhosts are much more likely to have future availability of 300 days or more.

One reason for this could be because new users will be classified as non-Superhosts, where they haven't had the chance to accept bookings for their properties resulting in large future availability.

However, Superhosts are more likely to have listings with availability ranging from 75–250 days. This could show that its beneficial to have Superhost status, but it's not particularly conclusive as for the very popular listings, with the availability of 50 days or less, it's more likely that they do not have Superhost status.
My Journey To Kaggle Competitions Grandmaster Status — Chapter the First
By: Daniel Benson
Tags: Kaggle, Data Science, Machine Learning, Awesomeness, My Journey

The coveted grandmaster status

In this series I will document my journey, from start to grand finale, to becoming a grandmaster on Kaggle.com. This will be no easy feat. Indeed, I anticipate both sweat and tears, but this arduous accolade will be well worth it in the end. Wish me the best of luck, though I assure you no one is wishing louder than myself.

The list of eye candy that Kaggle provides

To that end, I will first begin by explaining what the purpose of my “journey” is. Kaggle.com is in essence an exceptionally user-friendly site that caters to Data Scientists. Within its data-fied walls include basic courses for the aspiring Data Scientist, a Job Board for those a little further along, a discussion forum with fascinating insight, banter, and computer science witicisms on all topics Data Science related, a section for Data and software engineers to create and store their own jupyter-esque notebooks, a database filled with a diverse collection of data and datasets that would make any Data Scientist drool, and the creme de la creme of our attention, the Kaggle Competitions section, location of our coveted Grandmaster status.

Kaggle competitions consist of pre-selected datasets brimming with scrumptious data to use to any Data Scientist’s heart’s content. The challenge: solve pre-determined problems through the use of data cleaning, wrangling, engineering, analysis, and predictive modeling to achieve the highest accuracy score you can and climb (claw! struggle!) your way to the top of the leaderboards, netting performance tier medals along the way.

Kaggle’s performance tier is a tier system based on one’s performance on Kaggle’s many delicacies (surprise! surprise!). Each tier is also broken up into categories allowing a determined data enthusiast to earn different tiers in each of the different categories: obviously my interest of focus is in the competitions category. Tiers are ranked as follows: Novice, Contributor, Expert, Master, and Grandmaster, with tiers Novice and Contributor requirements each focusing solely on account registration and a “tutorial” checklist respectively. From there things begin to get more difficult.

Competitions expert tier requires 2 bronze medals to be won; competitions master tier requires 1 gold medal and 2 silver medals to be won; competitions grandmaster tier requires 5 gold medals and 1 solo gold medal to be won. These medals are awarded based on positioning in the aforementioned Kaggle competitions and can be found in the picture (curtousy of the wonderful Kaggle website itself) added beneath this paragraph.

My long-winded explanation finally complete, I leave you with this promise: future additions to this “Grandmaster Status” series, which will likely come in either weekly or bi-weekly installments, will include a much more palatable ratio of data/pretty-data-science-visualizations to Daniel-ramblings. I’ll end here by thanking you and bidding you adieu. Let the games begin.
Top 3 Reasons Why You Need A Financial Expert To Help You Secure Yourself After Retirement
By: Jeff Whatton
Tags: Retirement Plans, Retirement Planning, Retirement Strategy, Retirement

Retirement is defined as “The action or fact of leaving one’s job and ceasing to work.” Some people look forward to retirement. Others would not touch it with a ten-foot pole!

The “Golden years” mark the ending of routine work and the beginning of freedom. You can travel whenever you want to, sleep as much as you want and spend more time with your family than ever. The way it’s portrayed by society and social media has everyone thinking that retirement is a walk in the park.

Photo by Harli Marten on Unsplash

But that is not entirely true.

Retirement often comes with the looming threat of financial security. If you have a well-paying job or you’re the CEO of a thriving business, you have no worries. However, not everyone is that lucky. And this is why some people prefer to die without retirement — they have no assurance of financial security.

If you’re approaching the age of retirement and worried about financial security, the following three reasons will make the need for a financial expert clear:

They can help you customize a retirement strategy

The truth is that some people retire without knowing the options that their retirement fund provides. These options depend on the type of retirement plan you choose and the savings that you acquired while working.

Whether you’re employed by a company or self-employed, the help of a financial advisor is crucial.

Photo by James Hose Jr on Unsplash

A financial expert will help you navigate the details of your plan and chart an action plan for you to follow so that when you retire, you and your family can be at ease.

They have insight into unconventional methods that are tested and proven to work

This is the true value of an expert — having important knowledge not available to the masses. Your financial expert will help you put unconventional pieces in motion that will put you in a favorable position when you retire.

They breakdown complex concepts and give you a better understanding of your finances

There is a bible verse that reads, “For lack of knowledge my people perish.” This verse explains what happens to people that retire without the appropriate financial advice. There are concepts and jargon that you would not understand in their entirety unless you’ve been exposed to them.

Photo by Josh Hild on Unsplash

Financial advisers make everything easy to understand and will guide you to make the best decisions for your future.

Hiring a financial advisor is a task in itself — however, First Financial Group provides access to expert financial advice at affordable rates. They recently concluded a free lunchtime webinar and are ready to help you secure your future.
How I made a 30% ROI on Pandemic investments
By: Molly Patel
Tags: Money, Pandemic, Index Funds, Investing, Retirement

The second everything came crashing down March 16,2020 I, much like everyone else, did not know what would happen to all of my money. Do I hold on and save every dollar I can? Or invest it when rates are low and shoot my shot.

Well… I did the latter and it’s about to let me go to grad school- for free. How? INDEX FUNDS. These accounts let you invest in the total stock market or portions of it you just let compound interest do it’s thing.

I started by investing a good chunk of money in Vanguard’s VTI account the day it crashed to next to nothing. VTI is an ETF (exchange-traded fund) that encompasses over 3,500 individual shares, including the S&P 500. Historically, it has always done slightly better than the stock market so I thought to give it a try. Over time, I have been adding money in $500 increments whenever I had the extra cash. It’s always better to invest smaller chunks at different rates than a large chunk at one time. Your ROI will be higher, we can talk about that in a different article.

In the past 9 months, I was able to get a 30% return on the money I put in. 30% is pretty darn good. For those of you who are new to the investment world, real estate brings in about 6–8% a year (depending on where you live), the stock market gains about 10% and high -yield savings accounts only give you 2–3% back in interest. 30% is virtually unheard of. From March 2020 to now, I turned $3,570.00 to $4687.79. And this is just the start. If this trend continues, I can pay off my Master’s program with free money.

If I continue investing these small amounts, my money will triple in the next 10 years according to this chart below.

If you have the funds, I strongly urge you to chuck it in the Vanguard VTI. Continue to invest it in monthly, quarterly or any time it works for your budget. Consistency and patience is key. You’ll thank me when you are sitting in hundreds of thousands of dollars when you retire.