fabiochiu/medium-articles huggingface dataset:
How to create dashboard for free with Google Sheets and Chart.js
By: Rafał Rybnik
Tags: Data Science, Visualization, Dashboard, Web Development, Charts

Dashboards are a convenient method for quickly present data. For data scientists and analysts, it’s the tool of choice for showing results of their work to the non-technical recipients. Of course, there are many services for making online dashboards with no or little code. Using ready-made solutions such as Tableau, Qlikview or Google Data Studio, quick production of decent looking charts and dashboards is not a problem.

However, most of them are too expensive. We also often encounter situations where the possibilities for adapting views to exceptional data are limited. On the other hand, tailored, self-hosted solutions cause other difficulties. You need hosting and you are unlikely to want to play with the administration of a large system just to collect and present the necessary data.

However, it is possible to serve web applications directly from Google Apps Script.

In this article, I show you how to create a simple dashboard using data from Google Sheet and Chart.js. First, we will set up templated HTML in Google Apps Script. Next, we will with Chart.js. Then, you will learn how to fetch data from Google Sheets (e.g. when updating information visible in the dashboard) and insert it into the template. Once you set up the dataflow, you can use it in many projects.
Tiny graphics keep readers attention, and time does the same
By: Marius Popescu
Tags: Data Vizualisation, Infographics, Dashboard, Graphics, Tiny

Everyone in my family is born a bisect year. Three children and us, the parents!

I always get laughs and astonished looks when I tell this story to a friend. Still, there is no difference between a bisect year and the other years, right No year is special. Well, it depends.

We tend to recall milestones like marriage, babies and anniversaries. For your company, your products, you can also make a timeline of special years:

When was the company created

When was a product started

First customers

New versions

International versions, second and third products

Domain scientific publications

Standards and norms

In paralel with your product’s timeline, recall other important dates from the domain.

Context milestones complement your timeline and attract your customers

Speak to your customers by using events which are important both for you and for them:

- field events like congresses, discoveries or other publications

- national / international events

- new international standards (e.g. HL7 FHIR, ICD-11 or WHO ATC codes for medical apps)

Show these events in parallel with your timeline to give context and to connect your customers to your story.

Your turn now

What time references would you use in our tiny graphics?

Send your questions and ideas via the Tiny Graphics mailing list.

PS

My eldest daughter prefers saying we’re all born a Football World Cup year.
How to build a Dashboard in Python!
By: Jorge Silva
Tags: Data Visualization, Python, Data Science, Dashboard, Education

Let’s take a look at some examples hosted on the plotly gallery.

A Broker Dashboard with live prices

Broker Dashboard

An Uber rides tracking app

Uber Dashboard

The examples are self-explanatory, all the elements can interact with each other, rendering and updating dynamically.

There are many applications for Dash, especially if you are looking for quick implementations.

Let’s see how to code a simple dashboard in practice

The code will be divided into three different parts:

Initializing the App

Give it a layout

Start the server

Start by installing the necessary dependencies:

!pip install dash dash_core_components dash_html_components

Initializing the app

This step is creating a dash object that contains our main app

Give it a layout

We can use plain HTML elements by accessing the dash_html_components in a very intuitive way.

html.H1()

html.Div()

html.P() etc...

Our app content will live inside our main Div, it can contain as many elements as necessary, inside a list. It’s necessary to tell that HTML elements are not the only type of elements.

For more complex renderings such as graphs, drop-downs, input boxes we have to access dash_core_components.

In our basic example, we are going to write an introductory sentence inside our main Div.

Start the server

Set the flag “debug” to have a nice way to display errors on your dashboard, just remember to remove it before deploying it to production :)

The full code
Cloud Monitoring for GCE metrics [Basic]
By: Johanes Glenn
Tags: Gce, Dashboard, Monitoring, Stackdriver, Agent

Hi everyone to add one more context from the last story around logging, I want to visit the stack driver-agent for more elaborated metrics monitoring in Cloud monitoring.

It's one thing to collect our logs into Cloud Logging to be able to do queries and do a forensic operation as needed (please don't forget to add more logs on top of default Syslog). Now to add into the picture how we are able to collect metrics for our monitoring dashboard in stack driver? The answer is the stackdriver-agent for GCE (+ AWS EC can be included).

what to do:

Install the agent on the VM we want.

2. Check the authorization for the agent to write into monitoring

If we have follow the instruction correctly, we may see some help through the default dashboard in Cloud Monitoring.

Go to monitoring → Dashboard → use the default VM dashboard

In the image above we can see that this particular VM instance already installed with stackdriver-agent (the fluentd are not yet installed). After a while, we can see the metrics through the rest of the tabs (overview, CPU, memory, etc).
Torte statt Worte: Der neue Dashboard Designer
Tags: Runningtogether, Dashboard, Itwofm

CEO of RIB IMS | Digitize your business strategy — Next level real estate lifecycle and facility management. #iTWO #runningtogether | https://www.rib-ims.com
Key Metrics for preparing Dashboard for Clients
By: Zivanta Marketing
Tags: Digital Marketing, Analytics, Dashboard, Client Services, Dashboard Design

Irrespective of the type of business you are into, measuring Key Performing Indicators (KPI’s) for it is essential, but with dime a dozen KPI measures available online, it not only creates chaos but also deviates you from the path of strategic thinking and decision-making.

You don’t need to measure each and every KPI, as it is not only unnecessary but also creates a burden on your resources to successfully meet the fulfillment criteria for every KPI which is short of the target, as well as not necessarily you need every metric to understand how your marketing plan is performing. Generic KPI’s aren’t helpful in making sense for your clients and as well as whether they align with the type of marketing campaigns run by your team.

By employing revenue-focused marketing KPIs, you can ably show your team’s valuable efforts while staying aligned with your broad business goals.

Some Key Marketing KPI’s to show your marketing worth

Rate of Conversion — Individually measuring conversion rates for every phase of your marketing funnel shall lend you a productive insight into your customer’s journey. While studying that you may across certain loopholes in some phases, which can be a cause of you losing your potential revenue. For every organization, marketing funnels shall be different, and so shall your measuring metrics. It is important to have a well designed webpage to attract your customers and have your landing pages designed into conversion points. It is imperative to test the success of your landing page, so test them and refine them periodically, this is very important to ensure higher rate of conversion. Customer Lifetime Value (CLV or LTV) — It is important to calculate your customer lifetime value (CLV or LTV), to assess which customers carry value for you. To engage those customers, marketing campaigns have to be designed and adjusted; this can happen once we know which acquisition channels are giving the most profitable customers. CLV or LTV also helps in designing your retention marketing plan. Quality of Leads generated; Bounce rate and MQL:SQL ratio- Lead counts, the higher the number, the better it sounds and makes you feel good, but how are they helpful, if they don’t convert into closed deals, then it just shows a misdirected marketing strategy, as well as results in loss of money, time and resources. The main goal of any lead generation activity is to generate qualified leads, those who are in all probability going to indulge in transactional activity with your business. In this case, the Bounce rate can help you determine, whether you are attracting the right audience traffic to your website, whereas, MQL: SQL ratio acts as a litmus test for your sales team, wherein, whether they are successfully able to convert leads into proper customers.

These are definitely not all! But these metrics are undeniably the integral aspects of the performance dashboards, every client would like to see.

Read the Original Article: Vital Metrics for preparing Performance Dashboard for Clients
Solaris 11.4 SRU27 with Zones Sheet on the Dashboard
By: Marcel Hofstetter
Tags: Solariszones, Solaris, Dashboard

Solaris 11.4 GA was released in 08/2018. Since then Oracle published an update (SRU) each month.

We are now at SRU27 (November 2020). This new SRU contains a bunch of new features.

My favorite is the Zones Sheet where you can see how the Resource Usage of your zones is.

More details on my Blog

https://jomasoftmarcel.blogspot.com/2020/11/solaris-114-sru27-with-zones-sheet-on.html
Operational Reporting — Why is there still so much of it?
By: Andrew Bilsdon
Tags: Data, Dashboards, Andrewbilsdon

Operational Reporting - Why is there still so much of it?

The Analytics industry, in its various forms, consistently describes the exciting opportunities that exist in…
Explain Like I’m Five: Linear Regression in Machine Learning
By: Limas Jaya Akeh
Tags: Machine Learning, Linear Regression

Explain Like I’m Five: Linear Regression in Machine Learning

Photo by Jeswin Thomas on Unsplash

Linear Regression is one of the most traditional Machine Learning methods. To understand Linear Regression, just imagine it like this, “Will it rain or sunny today if the sky is cloudy, the weather forecast says that today will rain, and it is currently Summer, where rain is more unlikely”. We can define each event into independent features that correlate to the outcome — will it rain or not? And each event may affect the outcome much stronger than other events, for example, looking at the sky directly to check whether it will rain is much more likely and informative than looking at the calendar and knows that it is currently Summer. However, we assume that each of these events doesn’t affect the other, only the outcome.

Photo by Heidi Fin on Unsplash

Linear Regression tries to calculate how each feature independently affects the outcome (or the relationship between them). We can write the formula for Linear Regression as the following:

y = w0 + (w1 * x)

Where y is the output we’re trying to find, x is the input or feature that correlates to the outcome, w0 is the intercept, and w1 is the regression coefficient (or slope). Since we can know x and y from the data, we need to calculate the w0 and w1 ourselves.

Linear Regression Model by Wikipedia

Think of the regression coefficient as how important the feature is — the higher the (absolute) w1 value, then the output will be affected much greater. For example, if y is the chance of today will rain, and x is the humidity, if w1 is 1.05, then for every 1 point of x change, y will increase by 1.05. However, if w1 is 0.0001, then even with 1000 points of x change, y will increase by only 0.1.

The reverse also happens if the w1 value reaches the negative value, for example, if w1 is -1.15 then for every 1 point of feature (x) change, the predicted outcome (y) will decrease by 1.15 point, which means that there is a strong correlation between the feature and the outcome. The correlation will decrease as w1 gets near around the 0 value.

Correlation by SPH — Boston University

Alongside the regression coefficient, we also have intercepted, think intercept as the offset between the target and feature. A very simple example, if w1 is 1 (which means for every 1 point of x, then y will increase by 1), then x is 5, and then y is 10, then the offset between x and y is 5 (since y is supposed to be 5).

However, with the simple implementation, Linear Regression is rarely used for practical real-life problems, because the method is simple and does not account for many things such as uncertainty, the dependency between variables, and non-linear problems.
Scratch ile Basit Linear Regression
By: Akif Emre Çapoğlu
Tags: Scratch Programming, Data Science, Linear Regression, Deep Learning, Machine Learning

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
What makes a good Airbnb listing?
By: Tyler Lang
Tags: Machine Learning, Udacity, Linear Regression

For those of you listing your beautiful cottage house or the spare bedroom in your apartment on Airbnb, what will bring your listing success?

We all know ratings are a top priority for any type of vendor or service provider in obtaining a growing pool of clientele, but what is behind a good rating, especially for Airbnb, in which every listing is so unique in terms of its location, type of room/bed provided, price, cleaning fees, cancellation policies, and other specific client experiences. And just enough mediocre or bad reviews can really crush a listing’s success, while consistent and large-volume high ratings can let a listing sell itself without much work.

In the spirit of diving more into this large question, I decided to look at data from Seattle’s Airbnb listings from 2016. With just over about 3100 listings reported, there was enough data to test only a small number of the 36 features presented in the data, all of which may or may not play a part in determining a listing’s rating.

Before starting this analysis, I thought perhaps anything with a monetary value or related to the quality of amenities would contribute the most to a high rating. I decided to look at these features, along with features including host response time. Some features contained a lot of missing data, so in an effort to get as much information as I could out of the data, I created new features, or dummy variables, to indicate which rows had null values or missing data in a given column.

After cleaning up the data and applying a scikit-learn linear regression model, including and discluding new and existing features, changing the way in which nulls were filled by panda’s fillna function, the highest R-squared value I could seem to achieve was 0.039. In other words, this model cannot say much about the predictive value of each feature in terms of a rating, but still has some value in determining what TYPES of features seem the most salient.

As can be seen, to my surprise, the most important features seem to be the host response time and the property type. The lowest-predictive features are exactly what I had predicted to be more salient:

And finally, as can be seen, including dummy variables for any NaNs, or missing values per row, seemed to not add any value to the model, as they have little to no predictive power.

While this is just a surface-level analysis of the most salient features of what makes a listing highly rated, it is good motivation for future analyses that can look more into these more predictive features and see how they might affect one’s rating of a listing. NLP models might also be an interesting next step to determine the reasoning behind someone’s high or low rating.
Hi Andre, I liked the questions andread them with attention.I
By: Mohammad Ghasemi Hamed
Tags: Median, Linear Regression

Hi Andre, I liked the questions andread them with attention.I appreciate, because it helps us to keep fresh.

In the same line, I would like to add a comment on the 18th and 20th Questions:

18th Question: I am quite sure that you wanted to write

"independence of errors" but what we have here is the "dependence of errors". Besides the classical linear regression has five main assumptions where the independence of errors from the feature

variables and no autocorrelation of errors are of the main importance.

20th question: MSE is generally used for the mean approximation and MAE for the median.

Thank you for sharing your thoughts !

Feel free to add your response!
Simple Football Predictor: Week 15 Predictions
By: Nathaniel Selevan
Tags: Logistic Regression, Football, Linear Regression, Predictive Analytics, Tidbits

For once, I was running pretty ahead of schedule this week and was able to add a new feature to my excel file where I store my data. I can now see each team’s season average for each stat, and their rank in the league. So be prepared for some random tidbits the last few weeks of the season.

Week 14 Results:

In the win-loss game, last week might have been my worst performance of the season, shooting under 50% for the first time in a while at 7–9. I’m still a little baffled why the 60–65% range continues to perform so poorly, hopefully its just bad luck.

If you thought the win-loss went poorly, the spread went even worse with a disastrous 3–13. I am now 41–46–2 on the season.

Week 15 Predictions:

Predictions vs Spread (As of 5:48PM Thursday):

Raiders(-3.5) vs Chargers: Chargers

Broncos(+6.5) vs Bills: Broncos

Packers(-8.5) vs Panthers: Panthers

Rams(-17.5) vs Jets: Jets

Colts(-7.5) vs Texans: Texans

Titans(unavailable) vs Lions: Titans

Falcons(+5.5) vs Buccaneers: Falcons

Ravens(-13) vs Jaguars: Jaguars

Dolphins(-2) vs Patriots: Dolphins

Vikings(-3) vs Bears: Bears

Washington(+5.5) vs Seahawks: Washington

Cowboys(+3) vs 49ers: Cowboys

Cardinals(-6) vs Eagles: Eagles

Saints(+3.5) vs Chiefs: Saints

Giants(+5) vs Browns: Giants

Bengals(+12.5) vs Steelers: Bengals

Quick Observations:
A Look at the Maths Behind Linear Regression
By: Dhruva Krishnamurthy
Tags: Linear Regression, Deep Learning, Statistical Analysis, Machine Learning, Artificial Intelligence

In the diagram above, we have drawn an estimation line that is completely determined by its weights. To calculate the absolute error for one instance, we calculate the absolute difference between our datapoint and its vertical distance to the estimation line, which is its estimated value.

In order to convert this into a performance metric, we look at the squared error, which modifies the concept of the absolute error, turning it into a measure we can apply to our entire dataset. To calculate the squared error, we sum the squared differences of our estimated value and the actual value.

The usage of the square becomes very important later on when we look at algorithms that find the best weight combinations. Algorithms such as Gradient Descent, the most commonly used optimisation algorithm, requires the performance function to be differentiable and this squared error function satisfies that requirement.

So we have the squared error, which takes the idea of our absolute error and applies it to our entire dataset. We now need to turn this into a cost function that will serve as the objective of our model.

So we should remind ourselves what we are looking for in a cost function here and how this relates to linear regression. We want a function that provides us with an idea of how bad our model is in estimating our dataset. With this objective, we are then looking to minimise this function, as the smaller the value the “less-bad” our model is.

To produce this, we adapt our squared error into a mean squared error that helps us estimate our model across the entire dataset. We divide the squared error by 2n because of the presence of our squared term. Since 1/2 is a constant, we do not alter the information the cost function gives us by including this, but we have a cleaner result when differentiating this in weight optimisation algorithms.
Car Price Prediction
By: Aadil Aftab Shaik
Tags: Data Visualization, Data Science, Linear Regression, Machine Learning, Artificial Intelligence

This is my Linear Regression project; data analysis and prediction of car prices based on the data. I used Pandas, NumPy, Stats Models, Seaborn, and Scikit learn libraries to analyse the data and build the model.

I imported the dataset as a ‘data’ variable and analysed the data by using - ‘head’ and ‘transpose’ functions.

After analysing the data, I realized that there are many useless columns for our model, so I removed them by using a ‘drop’ function.

Now comes the cleaning of data. There are many ‘?’ data in our data frame, so I replace them with NaN values.

Thereafter I created a new column called ‘cylinders’, to store all the string values of ‘num_of_cylinders’ column as numbers, and converted all the string values of the remaining columns into float values by using an ‘astype’ function.

I replaced the NaN values of every column with their respective column medians and visualized the data with a pair plot, by using a ‘pair plot’ function from Seaborn library.

Pair Plot of all the data

I divided the data into an independent variable (x) and target variable (y). As my target is to predict the price, I took price as ‘y’ and remaining data as ‘x’.

I further divided ‘x’ and ‘y’ values into the train and test values with the help of ‘train_test_split’ module from Scikit learn, and took 25% of data as the test set, and remaining 75% as the training set.

I find the best fit line for the model by using a ‘fit’ function on a training set of ‘x’ and ‘y’ from Linear Regression (from Scikit learn).

I print all the coefficients (m) and intercept (c) of the model by using ‘coef_’ and ‘intercept_’ functions respectively.

M and C

You can check the score of the model by using ‘score’ function from Linear Regression.

Score

I use Stats Model library to further analyse and improve the model. In this library, the target variable (y) and independent variable (x) should be in the same data frame, so I concatenate (x, y) into one data frame.

I find the best fit line by using ‘ols’ function to assign a formula and using ‘fit’ function on it.

We can get coefficients (m) and intercept (c) of this best fit line by using a ‘params’ function.

M and C

And to get the detailed information we can use a ‘summary’ function.

Summary

CONCLUSION

I did all this to practice Linear Regression, and the following are all the things I learned while doing this project:

Learned how to decide the useless columns in order to remove them. Learned how to analyse data from a pair plot. Learned how to use Stats Models library to analyse and improve the model.

Project Github: Car Price Prediction
COUSERA Machine Learning Week09
By: Atsushi Ito
Tags: Linear Regression, Collaborative Filtering, Unsupervised Learning, Machine Learning, Anomaly Detection

atsss/cousera_ml

You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…
Least Squares in Linear Algebra
By: Ning Chen
Tags: Data Science, Linear Regression, Linear Algebra, Least Squares, Matrix

Linear regression is an important predictive analytical tool in the data scientist’s toolbox. In this blog, we implement least squares to approximate solutions of over-determined systems of linear equations by minimizing the sum of the squares of the errors in the equations. An introduction of how to use linear algebra to solve regression problems into machine learning and predictive analysis is reported.

View by Geometry

Linear equation has no solution when the matrix has more rows than columns, which means there are more equations than unknowns. Therefore, we cannot always get the error down to zero. However, a least squares solution can be accessed with optimization of error. In other words, we can get the exact solution when error is zero.

A toy example in the case of simple linear regression that approximates the linear mapping is presneted.

The column space of A is constructed by the combination of vectors a1 and a2. Due to b is outside that column space, the equation cannot be exactly solved for the vector x. For any vetor Ax, we have error Ax-b:

When vector Ax-p is reduced to zero, the error Ax-b is minimized. That is the case when Ax-b is perpendicular to the column space. Thereafter, we have the smallest possible error e=b-p, where p is the projection of b in space A. The ordinary least squares solution is

View by Linear Algebra

Now let’s rename the variables and solve the issue for a more general case. The model is composed by paired observations of the independent variable X and the dependent variable Y with error:

The matrix format is

By rearranging the equation, the error vetor with respect to 𝛽 is

Furthermore, the mean squared error (MSE) can be calculated and represented in matrix format:

Therefore, the gradient of the MSE with respect to 𝛽 is

Transpose the equation and set it to zero at the optimum of 𝛽, we have

The result is consistent with the conclusion derived above. Now let’s normalize the matrix products and estimate the coefficients.

which is the same as the results from the least squares of simple linear regression model.

The model can be extended to higher order as well,

and same optimum of 𝛽 can be calculated as follows.

The diagonal matrix can be significantly simplified by orthogonalizing the columns, which is the Gram-Schmidt process.

This simple and elegant formula works perfectly fine for the case of simple linear regression due to a limited number of computed dimensions, but with big data sets, ordinary least squares is computationally inefficient and expensive as it can potentially involve a huge number of complex mathematical operations. So in practice, gradient descent is performed to solve the linear regression optimization problem.
Is it possible to have a negative R square?
By: Akshita Chugh
Tags: Machine Learning, R Square, Linear Regression, Data Science

The coefficient of determination or R-squared represents the proportion of the variance in the dependent variable which is explained by the linear regression model. It is a scale-free score i.e. irrespective of the values being small or large, the value of R square will be less than one.

The Figure below is the graphical depiction of the coefficient of determination.

Negative R square is possible for the Linear Regression Models where fit is worse than the horizontal line. In the image below( see Figure 2) the residual sum of the square is more than the variance of Y hence ratio of the residual sum of the square to the variance of Y is greater than 1, therefore, leading to a negative R square.

The model in figure 2 doesn’t make any sense given the trend of the actual data points, and it is clearly a wrong model. However, is it possible to have a linear regression line similar to the one in figure 2 which doesn’t minimize the residual sum of squares to all the actual data points? A linear regression line like figure 2 would be possible if there are either constraint on the intercept or the slope.

For Example, if we want to fit a linear regression line that has to go through (0,3000) point then it would lead to a negative R square value.

Conclusion

R square can have a negative value when the model selected does not follow the trend of the data, therefore leading to a worse fit than the horizontal line. It is usually the case when there are constraints on either the intercept or the slope of the linear regression line.
Linear Regression 17 | Final Review
By: Adam Edelweiss
Tags: Slr, Model Diagnosis, Mlr, Linear Regression, Mathematics

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
What is Linear Regression ?!?!?!?!?!
By: Vikram Nayyar Cp
Tags: Data Science, Linear Regression, Machine Learning, Computer Science, Algorithms

Intro:

So in today’s post, we’re going to look into the Linear Regression algorithm. I’m aiming to break it down into simple terms and summarise it for all of you!

About:

Linear Regression — Models the relationship

Independent variables go on the X axis.

Dependant variables go on the Y axis.

If they both increase, that’s a positive relationship.

If as X increases, Y decreases, it’s a negative relationship.

It’s used to predict values for continuous variables.

Benefit:

Its simplicity is what stands out and makes it better than other ML algorithms.

Limitation:

It makes assumptions on the dependency between the 2 variables that can make it bias.

Final Things:

As always, thank you for making it this far, please leave comments, they help me so much. Be sure to like and share also, lets try and get 100 views within a week, like we have done previously. Follow CompAndCode on all platforms! Thank You Nibble Byte!!! Hashtags: #computerscience #coding #programming #technology #python #programmer #computer #tech #developer #coder #java #javascript #code #codinglife #webdeveloper #softwareengineer #engineering #machinelearning #softwaredeveloper #html #programmers #software #artificialintelligence #computerengineering #linux #datascience #cybersecurity #programmingmemes #hacking #bhfyp #informationtechnology #programminglife #hacker #css #pythonprogramming #php #webdesign #ai #science #programmerlife #codingmemes #deeplearning #developers #programmerhumor #coderlife #codingisfun #computers #kalilinux #developerlife #development #ethicalhacking #softwareengineering #computerprogramming #windows #coders #codingbootcamp #security #geek #daysofcode #programmingisfun #compandcode
Linear Regression Derivation -Math
By: Sri Vigneshwar Dj
Tags: Math, Linear Regression

So let’s see how to derive the a0 and a1 value from y = a0 + ai1*x , This is not a normal form derivation.

Here the matrices can be solver in any form
Simple Football Predictor: Week 13 Predictions
By: Nathaniel Selevan
Tags: Logistic Regression, NFL, Data Science, Linear Regression, Predictive Analytics

I have to admit that I’m a little relieved there was no Thursday night football this week. There was no way I was going to have these predictions ready by then. Its 11:30 Sunday morning and I’m just writing this up now. While the model wouldn’t have seen the results of a Thursday night game anyway, so the prediction would have been the same even if the outcome was already known, it still feels more proper to publish these predictions prior to game time.

Week 12 Results:

Win-Loss Model Results

I’ve now predicted 100 games exactly this season, and after a second straight strong week at 11–5, my season performance is now 58–42 or 58%, including 21–9 over the 2 week hot stretch.

On the season the lowest confidence interval is inching toward .500 and meeting expectations, while 55–60% continues to perform much better than expected. The 60–70% intervals continue to underperform, while the highest confidence interval continues to do great.

Against the spread I picked 8–8 last week, continuing a trend of picking very close to or at exactly 50% each week. My season performance is now 29–27–2, a record I don’t expect improved much as the season concludes, the model did have a Mean Absolute Error around 12.5 points, meaning each prediction is wrong by 12.5 points on average, so it more or less is random guessing.

Predictions:

Predictions vs Spread (As of 11:42 Sunday Morning):

Jets(+7.5) vs Raiders: Jets

Falcons(+3) vs Saints: Falcons

Bears(-3) vs Lions: Lions

Texans(+3) vs Colts: Texans

Titans(-4.5) vs Browns: Browns

Dolphins (-10.5) vs Bengals: Bengals

Vikings(-10) vs Jaguars: Jaguars

Cardinals(+3) vs Rams: Cardinals

Seahawks(-11) vs Giants: Giants

Packers(-8.5) vs Eagles: Eagles

Chargers(-2) vs Patriots: Patriots

Chiefs(-13.5) vs Broncos: Broncos

Steelers(-7) vs Washington: Washington

49ers (-1) vs Bills: Bills

Ravens(unavailable) vs Cowboys: Ravens

Quick Observations:

The Jets odds are almost dead even this week. I noted a couple of weeks ago that the 2 times my model predicted a Jets a victory they were leading at half time. This week with a 49.21% chance of victory, and a predicted margin of defeat of .03 points, I’m making the bold prediction that the Jets will be tied at halftime this week, before ultimately stumbling through the second half with less than a 100 yards total offense.

The model has brought back its strong home bias by predicting 12 home victories in 15 games.

I am currently enrolled in the Applied Data Science Professional Certificate program at the Thayer School of Engineering at Dartmouth College. Please reach out to met via LinkedIn (https://www.linkedin.com/in/nathanielselevan/) or comment below with all questions, comments, project suggestions, or to act as a mentor. All likes and comments will be well appreciated.
What is Linear Regression in Machine Learning
Tags: Python, Linear Regression, Machine Learning Ai, Machine Learning, Artificial Intelligence

What is Linear Regression in Machine Learning

In this short blog, I will go through what linear regression is and how it is applied in the field of machine learning. MLBot Dec 6, 2020·2 min read

Figure 1: Image of linear regression taken from Merchant Maverick

Linear Regression (LR):

Linear Regression is a statistical term that defines the relationship between two or more variables (one dependent and one or more independent) by using equation of line.

Y = mX+c

where m is the slope of line and c is y-intercept.

LR in Machine Learning(ML):

In ML, when you train your model then LR finds you the parameters that make the linear model fit best to your data.

It attempts to model the relationship between the variables by fitting a linear equation to the observed data. The line is fitted by either translation or rotation.

Y = MX + C

where m, slope controls the rotation of line and C, intercept controls the translation of line.

Figure 2: Image of fitting model taken from Machine Learning and Data Science

Another term that comes after that is Loss. Loss is the difference between the predicted value of a variable and the actual value.

For LR problems, one solution of loss is by using a cost function that measures the distance between the linear model’s predictions and the training examples; the objective is to minimize this distance.

References

Book by Aurelien Geron “Hands on Machine Learning with Scikit-Learn and TensorFlow”
Linear Regression 15 | Logistic Regression Inference
By: Adam Edelweiss
Tags: Mathematics, Statistical Inference, Linear Regression, Logistic Regression

Wald Test of Coefficient

(1) Hypotheses for Wald Test

Similar to SLR and MRL, to tell whether or not a predictor of logistic regression is significant or not, we have to test the parameter of this predictor. The hypotheses of the Wald test is,

(2) Assumption for Wald Test

The Wald test can only be valid when the sample size is large enough. If we assume that the logistic model is rely on a large sample size, then the MLE is of predicor p follows,

(3) Wald Test

Based on our discussion, we should conduct Z test for the parameter,

Here, we don’t Then, if

then we can reject H0 and reach the conclusion that the p predictor is significant.

(4) Confidence Interval based on Wald Test

The confidence level for an estimate

The confidence level for the odds ratio of an estimate

2. Likelihood Ratio Test

(1) Hypotheses for Likelihood Ratio Test

The likelihood ratio test is quite similar to a likelihood version F test with OLSE. Commonly. The null hypothesis and the alternate hypothesis are,

(2) The Definition of Deviance

We don’t have to fully understand this and we only have to know the meaning of this model in a general way. The deviance is defined as the difference of the likelihood value between the fitted model and the saturated model.

Then, because the second term of this equation is a constant, then the deviance can be simplified as,

Denote G² as the deviance of the model, then

then for the test above, define the test statistics as,

if,

then reject H0, the full model is significantly improved in the likelihood.

3. Deviance Residuals

If we want to evaluate the residuals for the logistic regression model, we have to modify the residuals because they are not the same as the OLS residuals. Basically, there are two commonly used ways for modification,

Pearson Residuals: this is similar to OLS studentized residuals

we should observe the person residuals to be distributed horizontally around the ei = 0 line.

Deviance Residuals

The deviance residuals are defined by,

The square of each deviance residual measures the contribution of each response to the deviance of the fitted model.

4. Pseudo R²

Because there’s no OLS principle for the logistic model, then we don’t have the regular R² for us to explain the variance of the model. But there’s actually some pseudo-R²’s that can suggest the goodness of fit for us.

Efron’s pseudo-R²

Mc Fadden’s pseudo-R²

See a more detailed explanation here (from UCLA).

To select the best logistic model, we can use the AIC or the BIC because these concepts hold for the logistic model.
Principal Component Analysis on LandClass: Australia Wildfires
By: Albert Um
Tags: Principal Component, Wildfires, Linear Regression, Loading

Photo by Joshua Hibbert on Unsplash

In light of the recent CallForCode Spot challenge: Australian wildfires, I will utilize PCA to analyze land classes and it’s relation with wildfires. The Australian Regions are:

- NSW=New South Wales

- NT = Northern Territory

- QL = Queensland

- SA = Southern Australia

- TA = Tasmania

- VI = Victoria

- WA = Western Australia

The land class dataset is a measurement (of 2015 alone) comprised of each region broken down into separate percentages of the terrain. There are 14 types of terrain that can describe what kind of landscape each state is.

The main difference between evergreen and deciduous is that deciduous shed all their leaves every autumn while evergreens retain some leaves. Herbaceous vegetations are plants that have non-woody stems.

Principal Component Analysis, in short, is a way to summarize a continuous dataset by creating new features that will explain the original variables. If I have 14 features (like above), then the maximum number of components derived will be 14. However, PCA can reduce a dataset to 4–5 features that will explain the majority of the variance. Also, PCA can help improve regression because the components created are not correlated with each other.

The created features can explain at least 95% of the variance from the original dataset. The transformed values in itself are not very interpretable, therefore I will now try to decypher each component by looking at its loading values. Loading values are calculated as below:

Loading values can help interpret principal components. For instance, PCA_1 can be interpreted as wooded marshlands as there are positive coefficients for Evergreen forests, wetlands, and bodies of water while negative coefficients for shrubs and herbaceous vegetation.

PCA_2 can be interpreted as outback-like regions with some water bodies. The positive coefficients on bare vegetation, water bodies, and lack of forests made me think of an outback scene with some batches of water.

PCA_3 may represent human civilized regions due to the positive coefficients on cropland and urban / built up.

And finally, PCA_4 to be park-like plains. In conclusion, principal components can be interpreted by calculating the loading values.
Simple Football Predictor: Week 12 Predictions
By: Nathaniel Selevan
Tags: Python, Logistic Regression, Linear Regression, Predictive Analytics, Scikit Learn

I have some good news, on Monday I will be starting a new job as a Data Quality Analyst, providing the opportunity to work professionally in Data for the first time. That coupled with the fact that my data science course is up to the capstone project phase (stay tuned in the next month for that as I switch focus to the NHL) means I don’t have as much time to dedicate to this project. I will still post weekly performance and projections, but will put improvements to the model on hold for at least a few weeks.

Week 11 Results:

Week 11 and Total Results

With an accuracy of 10/14, week 11 was our best week yet, and some of the losses were real close (Ravens in OT, Vikings in the last 2 minutes, etc) so this was nearly even better. Unfortunately, one of those losses, the Vikings vs the Cowboys ended my survivor season, but there’s always next year.

For the season we are still underperforming in some of intervals, but doing quite well in the 70+ and 55–60% intervals, and a record of 47–37 (55.95%) on the season, so we are doing a little better than random guessing.

Against the spread, the model had its best week yet at 9–5, bringing the season record up to 21–19–2.

Predictions:

Predictions vs Spread (As of 6:37PM Wednesday)

Lions (+3) vs Texans: Texans

Cowboys (-3) vs Washington: Washington

Jets (+7) vs Dolphins: Jets

Falcons (+3) vs Raiders: Falcons

Bills (-5.5) vs Chargers: Chargers

Bengals (+6) vs Giants: Bengals

Colts (-3.5) vs Titans: Colts

Jaguars (+6.5) vs Browns: Jaguars

Vikings (-4) vs Panthers: Vikings

Patriots (+2) vs Cardinals: Patriots

Steelers(unavailable) vs Ravens: Steelers

Broncos (+6) vs Saints: Saints

Buccaneers (+3.5) vs Chiefs: Chiefs

Packers (-9) vs Bears: Bears

Eagles (+6) vs Seahawks: Eagles

Quick Observations:

This will be the first time my model goes up against a 16 game week, up until it has only predicted 14 game weeks, hopefully 10 win weeks become the new normal.

At a 50.04%/49.96% split, the Cowboys-Washington matchup might have the closest odds we’ve seen all season.

The Jets are the home team, but not listed as the favorite, so I guess there will be no halftime lead this week.

At 73.42%, the Steelers are the most favorite team this week, a testament to how the Ravens have struggled a bit recently.

I am currently enrolled in the Applied Data Science Professional Certificate program at the Thayer School of Engineering at Dartmouth College. Please reach out to met via LinkedIn (https://www.linkedin.com/in/nathanielselevan/) or comment below with all questions, comments, project suggestions, or to act as a mentor. All likes and comments will be well appreciated.
Linear regression and gradient descent for absolute beginners
By: Lily Chen
Tags: Data Science, Linear Regression, Gradient Descent, Getting Started, Machine Learning

Linear regression and gradient descent for absolute beginners

Image created by me

Let’s say we have a fictional dataset of pairs of variables, a mother and her daughter’s heights:

height of mother/daughter pairs

Given a new mother height, 63, how do we predict* her daughter’s height?

The way you do it is by linear regression.

First, you find the line of best fit. Then you use that line for your prediction*.

*Note: I used “predict/prediction” in this article. However, a reader pointed out in the comment below that the correct terminology is “estimate/estimation.”

Line of best fit, or “regression line”

Linear regression is about finding the line of best fit for a dataset. This line can then be used to make predictions.

How do you find the line of best fit?

This is where gradient descent comes in.

Gradient descent is a tool to arrive at the line of best fit

Before we dig into gradient descent, let’s first look at another way of computing the line of best fit.

Statistics way of computing line of best fit:

A line can be represented by the formula: y = mx + b .

The formula for slope m of the regression line is:

m = r * (SD of y / SD of x)

Translation: correlation coefficient between x and y values ( r ), multiplied by the standard deviation of y values ( SD of y ) divided by standard deviation of x values ( SD of x ).

The standard deviation of mothers’ heights in the data above is approximately 4.07. The standard deviation of daughters’ heights is approximately 5.5. The correlation coefficient between these two sets of variable is about 0.89.

So the line of best fit, or regression line is:

y = 0.89*(5.5 / 4.07)x + b

y = 1.2x + b

We know that the regression line crosses the point of averages, so one point on the line is (average of x values, average of y values) , or (63.5, 63.33)

63.33 = 1.2*63.5 + b

b = -12.87

Therefore, the regression line as calculated using correlation coefficient and standard deviations is approximately:

y = 1.2x - 12.87
Linear Regression with a Graph Database
By: Shreya Chaudhary
Tags: Linear Regression, Pokemon, Plotly, Tigergraph, Plotly Express

Linear Regression with a Graph Database

(Note: This is a bonus blog based on a series. To obtain the data used, refer to the past blogs.)

Hello! Today, we’re going to learn how to run a linear regression using TigerGraph. In this example, we’ll be running a linear regression with the Pokémon’s height and weight. Let’s get to it!

Part I: Setup

To start, we’re going to need to import the necessary libraries and create a connection to our graph.

The libraries we’ll be using are pyTigerGraph, Plotly Express, and Pandas. You can import them using the following:

import pyTigerGraph as tg import plotly.express as px import pandas as pd

Note that if you are using a Colab notebook, you’ll need to install pyTigerGraph first:

!pip install pyTigerGraph

Now that you have your libraries installed, start your solution on TigerGraph Cloud and create a connection in Python.

conn = tg.TigerGraphConnection(host="https://SUBDOMAIN.i.tgcloud.io", password="tigergraph", gsqlVersion="3.0.5", useCert=True) conn.graphname = "PokemonGraph" conn.apiToken = conn.getToken(conn.createSecret())

Step II: Write a Query

Next, write a query to grab all the Pokémon. Here, we’ll just create a seed with all the Pokemon vertices and use that. We’ll assign the results of the query to a variable called result .

result = conn.runInterpretedQuery(''' INTERPRET QUERY () FOR GRAPH PokemonGraph { catchThemAll = {Pokemon.*}; # Pokemon.* means that it'll grab all the Pokemon PRINT catchThemAll; # print displays the results } ''')

Step III: Create a Dataframe

To easily plot the data into Plotly, we’re going to need to format our results into a dataframe. Here’s how we’d do that:

result = result[0]["catchThemAll"] # Grabbing just the catchThemAll result # The lists of what we need

height = []

weight = [] for i in result: # For each pokémon

height.append(i["attributes"]["height"]) # Add the height

weight.append(i["attributes"]["weight"]) # Add the weight d = {'height': height, 'weight': weight} # Put it in a dictionary df = pd.DataFrame(data=d) # Put it in a dataframe

Step IV: Plot it on a Scatter Plot!

Finally, plotting everything is as simple as two lines:

fig = px.scatter(df, x="height", y="weight", trendline="ols") fig.show()

And with that, you should be done!

Linear Regression, Image by author

Step V: Explore and Experiment!

That’s it! You’ve now created a linear regression using data from a graph database! Try to query for other variables and run a linear regression using those variables or enhance the graph, adding a title, changing the colours, etc. Good luck!

You can also reference the Colab here: https://colab.research.google.com/drive/1U1mG7gzce3lCLhrsi0KCUZn3z0v-iLca