fabiochiu/medium-articles huggingface dataset:
First steps with GraphQL
By: Алексей Колмаков
Tags: GraphQL, Crypto, Graph

https://thegraph.com is a decentralized protocol for indexing and querying data from blockchains.

Right now, it works with Ethereum only but there are plans to implement it to another blockchains. The Graph is invented to help dApps to query data which could not be simply obtained using blockchain node.

Queries could be done using GraphQL API. There is Playgrounds feature in every subgraph presented in explorer https://thegraph.com/explorer/. You can try to execute queries using GraphQL API. I would recommend to check Uniswap V2 subgraph firstly: https://thegraph.com/explorer/subgraph/uniswap/uniswap-v2. It was provided as example in Curators Kickoff Stage 2 https://www.youtube.com/watch?v=noH_33MsXQc and you should watch.

Let us try to execute some queries in Uniswap subgraph.

There is example:

It should output: the first five uniswapFactories and its fields: id, pairCount, pairs, and the first five tokens and its fields: id, symbol, name, decimals. But if we try to execute it, we get the error

Yes, example is incorrect! Most likely the team will fix it soon. What is the problem? On the right side you can see the schema of the subgraph

Let’s click on UniswapFactory:

We see entities of UniswapFactory and as you can see there is no field ‘pairs’. But we asked subgraph to output this entity for UniswapFactory. That is the problem. If we try to execute the query:

we see the valid result:

So, in common case if you want to query something using subgraph, first you should look at subgraph’s schema and its entities. Take entity interesting for you e.g. ‘token’ and add it to the query schema. If you want to query a single token, you use ‘token’, but if you want to query all token entities you should use ‘tokens’.

Further I provide more complicated example of the query:

In ‘pairs’ entities query we see:

skip: 10 — means that we ask to skip the first 10 entities

first: 5 — as we know it means that we ask to output only the first 5 entities of collection (but after skipped ones)

where — is filter and in {} we define its condition, in our case reserve ETH should be more than 100000 (_gt — prefix which means ‘greater than’, and there are other prefixes which you can see in the docs)

In ‘tokens‘ we see interesting condition for filtering: ‘symbol_in: [“USDT”,”USDC”,”USDT”]’, prefix _in means that its entity should be equal to one of the values defined in the array […]

That’s all! I hope this tutorial would be useful for you. Go on Astronauts!
Should you include a Homepage Hero Carousel ?
By: Juan Diego Rodriguez
Tags: Web Design, User Experience Design, User Research, UX

Should you include a Homepage Hero Carousel?

The Homepage Hero Carousel can be used in pretty much every website no matter the industry or business. This feature can be added to any single client or personal site even when the research shows that visitors only care or spend time in the first Carousel slide, the rest of them, they are not engaged at all.

If you are lacking resources or budget, you will not be able to follow a proper design for an effective Homepage Hero Carousel. Additionally, the site will have little information architecture. For this reason, I believe it is not best to include a Carousel, however, if you still want to pursue designing a Carousel here are some suggestions I found in the Nielsen Norman Group:

1. Select the number of Carousels, normally they recommend a max of 5.

2. Properly indicate the Carousel slide information (aka number, sign, colour, photo).

3. Add the content in the secondary Carousel slides as part of your IA in another place within the site

4. Define the timing and effects of the Carousel

5. Design Carousel effect and responsiveness for mobile devices

If you are looking for your website visitor to focus on a key message, and again low budget it will be better to have a Hero Image only instead of a Homepage Hero Carousel. As described in the points above, all those tasks require time, money and effort, and if you want to have the visitor spend the right amount of time on the site to collect the information needed do not spend time in some extra content you would like to show. In this case, not having a homepage carousel will be the best approach.

In the research: “Carousel Usability: Designing an Effective UI for Websites with Content Overload” done by Nielsen Norman Group, they went over the best practices of implementing a Homepage Hero Carousel. They suggested that creating a Homepage Hero Carousel can be problematic and confusing if you are not following the best practices.

Another research online: “Carousel Interaction Stats” by Erik Runyon, talks about an experiment analyzing the clicks generated on an Auto-forwarding and a Static Homepage Hero Carousels were in two different websites the end majority of clicks completed wherein the first Carousel slide. There were some differences in the number of clicks by visitors completed in the following Auto-forwarding slide compared with the Static carousel, but overall the engagement is only in the first image, having multiple slides will not give your visitor more engagement.

Implementing a Home Page Hero Carousel properly cost money and makes the site development longer, this is why it is not recommended unless proper research, planning and design is completed following the best practices from information available online.

I hope you have enjoyed this article, please feel free to leave any comments below

Do you want to know more about my UX journey? Read my other post The UX principle that will improve your Kanban board
Less is more? The paradox of choice — Behavioural economics in UX
By: Przemek Vincent Szustak
Tags: User Experience, Behavior, Behavioral Economics, User Experience Design, UX

Less is more? The paradox of choice — Behavioural economics in UX

People often think more means more. Often, however, the opposite is true: the more choices we have, the higher our expectations become — and the more we worry about making the ‘wrong’ decision. Enter the ‘Paradox of Choice’. Przemek Vincent Szustak Mar 3, 2020·4 min read

An interesting study was published in 2000 by psychologists Sheena Iyengar and Mark Lepper from Columbia and Stanford University. On a regular day at a local food market, people would find a display table with 24 different kinds of jams. Then on another day, at that same food market, people were given only 6 different types of jam choices.

Which display table lead to more sales?

With fewer options, 40% of the shoppers tried the jams and 30% made a purchase. With more options, 60% tried the jams, but just 3% made a purchase!

The conclusion: “choice is alluring but confusing”. Customers given too many choices are ten times less likely to buy!

What does this mean?

Keep things simple, limit options and keep testing.

As we can see, when we’re given more options, we are less likely to choose. Why? Because we suffer from ‘choice paralysis’.

More choice requires more time and effort. This can lead to anxiety, stress, unhappiness or even regret if a poor choice is made. It’s hard and it’s difficult to make a good decision when you’re overwhelmed with information and options. You can’t process it effectively.

Instead of the risk of making a poor choice, we choose not to make a choice at all. No action is taken when the cognitive effort to compare all the options is too great.

Too much choice = no choice at all

Less choice = more satisfaction
Heuristics in Nature
By: Udaya Kumar Sivagurunathan
Tags: Usability, User Experience, User Experience Design, UX, Heuristics

When I begin my career as a UX Designer, Heuristic is the word which kept revolving inside my mind.

I happened to see all the 10 heuristics proposed by Jakob Nielsen. It took me some time to understand what it really is. Even today I find fresh UXers facing the same difficulty. It is all because of the template what we happen to see and evaluate about Heuristics. The template design has given an odd feel to it. Finally, my mentor (Mr. Kingsley Sathish Kumar of UX Mint) helped me to understand heuristics with respect to nature. His insights on this made me understand and remember them easily.

Are you exited to know about this?

I will give you a short insight on this. Before we begin, think about nature, how beautiful it is, at the same time how very dangerous. Remember Tsunami, how many lives we lost? South Asia never faced such a natural disaster in past 5 decades.

What was the reason behind that? It was because of tremors at Sumatra Island. That is one of the many reasons behind it. As we dig deeper into that, we will be able to find the root cause.

So what is the root cause of Tsunami in South Asia?

One of the root causes on why Tsunami had such a disastrous impact in South India is that, “N” numbers of coral reef found in Indian Ocean, fishes and other sea species lives near these coral reefs. During the time of fishing, fishermen bomb coral reefs to get bunch of fishes. They have done this and due to this loss to the nature, Tsunami made its way to the land. So what is the relationship between Tsunami and Coral reef? This question triggers lot of sense.

OK!! Get into the concept of heuristics.

Coral reefs are the ones which prevents sea from erosion and some disasters like Tsunami. We can connect this with Error prevention topic in Heuristics. Nature knows about its error (disaster) and made coral reefs to prevent it.

Other examples observed in our day to day life for Error prevention are: Calamus beads which have medicinal value tied in infants hand, they lick and chew it and this prevents Gastritis. Side stand indicator in two wheeler. Plug points enabled with block inside it.

For visibility of the system status. How do we find out whether it is day or night? How do we decide which is which? When there is Sun, we call it as day. That is visibility.

Likewise we are able to explore various examples in nature for other Heuristics too.

Why should we do it?

It is to remember and recollect Heuristics on the longer run. Somewhere, sometime these observations would reflect in our design. That’s fruitful.

Say cheese :D
Mobile App User Experience Design Case Study by Mesrop Janoyan
By: Mesrop Janoyan
Tags: Rapid Prototyping, User Experience Design, Product Design, Wireframe Design, User Interface Design

Project Brief

Ignite is a multiplatform VPN venture by engage:BDR, a cross-device display and video advertising company based in West Hollywood, California. As a virtual private network service reseller, Ignite seeks to provide online privacy and anonymity by creating a private network from a public internet connection using a white-label VPN implementation.

Project Objectives

The primary objective was to design an engaging, trustworthy, and memorable user experience on iOS through a white-label VPN API and market it to consumers who are growing increasingly aware of online cybersecurity and data privacy risks. The iOS experience and visual design were subsequently adapted to Android, macOS, and Windows platforms.

My Role

Over 11 months, I served as Ignite VPN’s Principal and sole UX/UI Designer. During that time, I designed Ignite’s user experience, user interface, and interaction design on mobile, desktop, and web platforms, as well as brand identity and style guide.

Project Overview: Product Design Sprints

I conducted the product design portion of the project in a series of five iterative design sprints to answer crucial product questions through data-driven design, prototyping, and testing. Each sprint was designed as a microcosm of the overall design thinking process employed in this project.
Relationship goals in Travel Industry — UX Case Study of Designing Cancellation Policies
By: Isha Beniwal
Tags: Travel Design, User Experience Design, Interaction Design, Case Study, UX Design

In the early 2020s, COVID-19 struck the world. The travel sector had to face an angry horde of anxious customers wanting to change and cancel their bookings. The load on the customer care teams was immense. The pandemic blindsided us and thrusted upon us numerous complex issues, for which there were no immediate fixes. But the MakeMyTrip team has left no stone unturned in trying to find common ground and makes this experience hassle-free for the consumers while backing our suppliers.

It became imperative for us, as a team, to remain clear and honest with our users, so that they don’t feel cheated. Think about relationships. To avoid an ugly break up, it’s best to be upfront and honest with each other. In the same way, as designers, we need to be transparent with our user. It’s natural to feel cheated if one doesn’t know the entire deal before diving in.

Being transparent with the cancellation policy is the best policy

Beware the angry customer — they can go on a rampage ranging from dialling customer support, complaining on social media, down-rating your app, and creating an overall bad impression of your brand on social media and other channels.

MMT’s Holidays line of business has its own complexity as the packages are created with a combination of Flights, Hotels, Activities and transfers. Each inclusion comes with its own cancellation nuances.

Which is why it’s important that the users can easily find your Cancellation Policy on the website or app. We had to make sure it’s easy to locate and understand.

Defining the problem

For any relationship to work, communication is key. Traditionally, cancellation policies came in long, static, textual, bullet-point formats where everything looks the same.

The aim was to clearly highlight cancellation policy while giving the user an easy way to compare between plans; thereby improving the overall funnel hygiene.

Sometimes an amicable divorce rests on the shoulders of a solid pre-nuptial agreement. Before you break up, it’s important to talk about who keeps the dog and who gets the Netflix account. Before you book with MakeMyTrip, you should know what you’ll be getting back.

Objectives

Display different kinds of protection plans that a user could pick from

Pre-define and convey the difference between date change and cancellation to avoid confusion

Clearly highlight the default plan selected

Process

Defining structure:

As the components were many we decided to have dates as a fixed element to help users compare between the policies and protection plans.

Exploration:

Experimented with different ways of approaching the design e.g. Rate plan, Timeline view, Tabular representation etc
Notion iOS App Redesign — Zero Cost UX Case Study for the app for taking notes
By: Helen Gorbachenko
Tags: Ux Case Study, User Experience Research, User Experience Design, Ux Design Process, UX Design

Notion iOS App Redesign — Zero Cost UX Case Study for the app for taking notes Helen Gorbachenko Feb 6, 2020·10 min read

Perfecting the Text Editing

Introduction

The Notion is an app for managing plans, records, and to-do’s in one place. I used to have weekly notes in it.

By the time I am using the app, I noticed UX issues in the text editing and adding pages. I thought, what if such problems have not only me but other users.

The purpose of this case study is to showcase the possibility of Zero Cost UX Research and to present an improved UX according to users’ feedback.

UX Activities: User Interviews, Secondary Research, Competitive Analysis, User Persona, Sketches, Wireframes, Paper Prototypes, User Interface Design, Usability Testing.

User Interviews, Secondary Research, Competitive Analysis, User Persona, Sketches, Wireframes, Paper Prototypes, User Interface Design, Usability Testing. Used Tools: Sketchbook, Notes app, UserBit, Twitter, Sketch app, Principle.

Sketchbook, Notes app, UserBit, Twitter, Sketch app, Principle. Scope: The whole process took me one month of work for evenings and weekends.

Problem Space

Judging from my own experience, I assume that the Notion app has the following area for improvements: ‘Page Lock’ feature for selecting the text, the ‘Undo’ action — especially in its behavior in adding pages, lack of duplicating rows in the mobile app.

The ‘Page Lock’ feature makes me as a user to do several actions to achieve the selection of several text rows. ‘Undo’ does not work with the accidental adding of a page, though it could save me a couple of munites. The biggest struggle I have with the duplication of checkboxes from the mobile — for this, I need to create a new row and convert it manually every time.

To validate my hypothesis, I have conducted several research activities.
Ecommerce Bakery & Cakes — UX Case Study
By: M Renton
Tags: User Experience Design, User Research, Ecommerce, Case Study

The emergence of online food delivery services such as UberEats, Foodora, and DoorDash has fundamentally changed the way consumers are purchasing food. Despite the eCommerce industry experiencing double digital growth year over year, there are still many products that are not traditionally viewed as items that are sold online. Cakes and bakery goods are one of the non-traditional items.

Myself and 3 other team members were tasked to conduct user research and deliver mid-fi prototypes for an eCommerce store that sold and delivered bakery and cake goods.

The Challenge:

We wanted to create an excellent buying experience that plays to the strength of online shopping while overcoming consumers’ hesitations about purchasing cakes and bakery goods online. Bakery and cake goods also pose their own unique challenges and the need to stand out from an extremely competitive industry became very apparent.

Research Phase:

Domain & Organizational Research:

To start, we did extensive domain and organizational research to get a better understanding of the industry as a whole. This allowed us to set our expectations and more effectively ask the right questions in our interviews and survey. We learned about the current status of the industry, its projected growth, and how consumer preferences are changing.

Interviews & Surveys:

Armed with a host of questions from previous research, we set out to interview 4 people that we felt would be good candidates to use an eCommerce platform we were designing. As we started to compile the research some trends and insights started to emerge.

Interview Findings:

Many interviewees never considered purchasing these goods online.

They were time-strapped and welcomed the idea that online shopping could bring.

They had objections and trust concerns about product quality and delivery.

organic ingredients were important factors with some interviewees stating that adhering to dietary requirements would help them make a purchase decision online.

From there we created a 13 question survey to paint a better picture of who our potential user is and what challenges they face. In total, we received 62 responses from our survey. The results reinforced our findings from our interviews. Despite these challenges, we found some opportunities and gaps in the market that an eCommerce store could potentially solve.

Survey Findings:
Beginner’s Guide to Creating the SVD Recommender System
By: Mayukh Bhattacharyya
Tags: Numpy, Recommender Systems, Data Science, Pandas, Collaborative Filtering

Utility Matrix

The dataset in the current form is of no use to us. In order to use the data for the recommender engine, we need to transform the dataset into a form called a utility matrix. We make a function create_utility_matrix in a new script. Name it recsys.py . We shall use the functions in this script to work on our train and test set.

As a parameter, we pass a dictionary that stores the key-value pairs for each column of the dataset `data` that we are also passing. From the dataset, we’ll see the column number or column names for each of the corresponding fields, the column userId or column 0 for the key ‘user’, column movieId or column 1 for key ‘item’ and column ratings or column 2 for key ‘value’.

Utility Matrix is nothing but a 2D matrix where one axis belongs to the users and the other axis belongs to the items (movies in this case). So the value at (i,j) location of the matrix will be the rating that user i gave for movie j.

Let’s give an example to clear up a bit more. Suppose we have this dataset of 5 ratings.

+----+----------+-----------+----------+

| | userId | movieId | rating |

|----+----------+-----------+----------+

| 0 | mark| movie1| 5 |

| 1 | lucy| movie2| 2 |

| 2 | mark| movie3| 3 |

| 3 | shane| movie2| 1 |

| 4 | lisa| movie3| 4 |

+----+----------+-----------+----------+

If we pass this dataset through the create_utility_matrix function described below, it will return an utility matrix like this and auxiliary dictionaries of user_index and item_index as shown below.

+----+----+----+

| 5 | nan| 3 | # user_index = {mark: 0, lucy:1, shane:2, lisa:3}

+----+----+----+ # item_index = {movie1:0, movie2: 1, movie3:2}

| nan| 2 | nan|

+----+----+----+

| nan| 1 | nan| # The nan values are for user-item combinations

+----+----+----+ # where the ratings are unavailable.

| nan| nan| 4 |

+----+----+----+

Let’s see the function now.
Will NumPy become Python?
By: Emmett Boudreau
Tags: Numpy, Mathematics, Programming, Data Science, Python

As Python becomes more and more of a statistical language, it will be interesting to see if Numpy makes its way into the base language. Obviously Python’s use in statistics will likely influence the likely-hood of more functional features leaking into the language. Though Numpy will certainly still be providing a lot of heavy lifting and tools, I can’t imagine the base language not taking a few pointers and Numpy doing a few deprecation. Though, with a tool like Numpy, it’s likely on many machines already, it does feel a little segmented and odd to need a package to do array multiplication in one line.

Though whether or not some features of Numpy are crunched into the Python base isn’t that big of a deal, and Numpy is certainly seen as an extension of Python, I do think that the concept is interesting. It is in fact very possible, and makes since. While conforming to the idea is plausible, there are also good arguments to abstain. As Python becomes more widely used for Data Science especially, we might see some features that cater to statistics.
Simple Linear Regression with an example using NumPy
By: Arun Ramji Shanmugam
Tags: Numpy, Linear Regression, Machine Learning

What is Linear Regression ?

Linear regression is the mathematical technique to guess the future outputs based on the past data .

For example, let’s say you are watching your favourite player playing football in today’s match , he is having very good track record against this opponent team with an average of 2 goals in every match , based on this simple calculation in your mind you may expect him to score at least 2 score or more than that , so what your brain did was calculating the simple average or mean.

average = total score against opponent team / number of match against opponent

Linear regression also similar to that but instead of taking an average, we are doing much better statistical guess using linear relationship between the input variable (x) and target variable (y) .

Note : Linear Regression can be applied only for continuous variable like rain vs humidity , heart rate vs running speed etc .

let’ see how to it works by implementing it in popular numerical computing python package NumPy .
Coding four-layer neural network
By: Abdallah Chamakh
Tags: Neural Networks, Numpy, AI, Machine Learning

in my last post, I explained how the neural network in detail and I suggest to check it if you don’t have any idea about how neural network works otherwise you could skip it, however in this post we will code 4 layers neural network with just python and numpy .

what is the function that we will approximate

in this post, we will approximate the exclusive or function which outputs 1 when two inputs are similar and 0, when they are different, let’s take an example for that

0 , 0 => 1 | 1,1 => 1 | 1,0 => 0 | 0,1 => 0

import numpy and set hyperparameters :

generate data and initialize both weights and biases :

create the training data which is simply the true table of the Xor function :

initialize randomly the weights and biases from a normal probability distribution

create the activation function

sigmoid function

sigmoid function prime

write the train function

in this function, we have both the feedforward and backpropagation algorithm

if there is any concept I want you to leave with it from this post is backpropagation which is simply is calculating the partial derivative of the cost function respect to each weight and bias in the neural network and this value gives us a clear direction to how we should updating our parameters to serve the purpose of decreasing the cost function.

now time to train our model for 1000 epochs

It’s time to make a prediction !!

Finally that you get value from this post and stay tuned for more AI content
Lambda With Numpy The Easy Way. Lambda python environment is bare bone…
By: Jean-Frederic Plante
Tags: Lambda, Numpy, AWS, Python

Photo by Stefan Steinbauer on Unsplash

Lambda python environment is bare bone by defaults. For my project creating math worksheets for kids, I needed a couple of additional packages such as numpy and fpdf.

AWS online instructions are usually very complete, and this guide on creating a Lambda deployment package is no exception. Other great online resources include this post.

Homebrewing headaches

However, when installing numpy via pip on my mac, I get this cryptic error upon running my function.

[ERROR] Runtime.ImportModuleError: Unable to import module 'your_module': IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! Importing the numpy c-extensions failed.

There is a documented exception for HomeBrew python configurations which is how I upgraded my macOS Mojave python to 3.7. In the case of numpy, we can see pip looks for a macOS specific wheel package and binaries.

https://files.pythonhosted.org/packages/8d/4b/b6339340169862935ef5757db7e5869af7576f03148d069869edbd523ef2/numpy-1.17.1-cp37-cp37m-macosx_10_9_x86_64.whl

We need the linux wheel package for hosting our code on AWS Lambda.

A shortcut, the manual way

While the correct way would be to leverage a docker image to generate your package directory, here is an easier way if you’re in a hurry.

1- Create a directory where you will store your package dependency. Following the conventions in this guide, let’s create a directory package

2- On Pypi, find the linux deployment package for the version of python you want to use on Lambda. As an exampe for nympy, navigate to https://pypi.org/project/numpy/, Download files and find the x86 manylinux distribution for your python version.

choosing the many linux x86 distribution for Python 3.7

3- Download the whl package to your package directory and unzip it (the wget link should come from the download link from pypi)

wget https://files.pythonhosted.org/packages/25/eb/4ecf6b13897391cb07a4231e9d9c671b55dfbbf6f4a514a1a0c594f2d8d9/numpy-1.17.1-cp37-cp37m-manylinux1_x86_64.whl unzip numpy-1.17.1-cp37-cp37m-manylinux1_x86_64.whl

rm numpy-1.17.1-cp37-cp37m-manylinux1_x86_64.whl

4- Rinse and repeat for other packages.

5- Zip and push. For this I created a small upload.sh script with the following:
Convolutional Neural Networks and Backpropagation using Numpy
By: Kevin Maciver
Tags: Convolutional Network, Numpy, Machine Learning, Backpropagation

Okay! Let’s do this..

So, I prepared this story to try to model a Convolutional Neural Network and updated it via backpropagation only using numpy.

The idea behind all of this is to further understand backpropagation.

Before I begin I’ll like to disclaim the following statements:

1. I do not intend to built the most accurate model at this moment since the idea is to unravel backpropagation.

2. Running all this will consume a lot of your memory and it will take a long time, mainly because i wanted to be more didactical with my coding and use A LOT of for loops. So I apologise for not being to efficient, but that was not really my goal in this kernel.

For this story I’m going to use the old and famous MNIST data (the hello world of Neural Networks)

Now that the data has been loaded and transformed it’s time to model our Neural Network.

For this excercise I’m going to use the following architecture:

Defining function for calculating Convolutional foward passes

Defining function for calculating Maxpool foward passes

Defining function for calculating flatten foward passes

Defining function for calculating Dense foward passes

Defining functions for calculating final outputs foward passes

Defining Calculation for Loss Function

Okaayy!

Now, for those who are still with me here comes the tricky part.

We now have defined all the functions we are going to need to do a single foward pass in our CNN model.

In order now for our model to learn, we need to define the backpropagation functions.

As you may know, backpropagation uses partial derivatives and gradient descent to update the parameters (weights and biases) in order to minimize the loss and therefore learn.

If you have doubts about the subject, I encourage you to check out the materials of 3blue1brown, Siraj Raval and Andrew Ng on youtube.

Output Layer Backpropagation

Flatten Layer Backpropagation

Maxpool Layer Backpropagation

Convolutional Layer Backpropagation

Well, the worst part (or best if you like this stuff ) is over. Now that all the functions have been declared we can beggin training our data.

To do that we must define a new function.. the mother of all functions..

“One Funtion to rule them all, One Funtion to find them,

One Function to bring them all and in the darkness bind them”

Okay.. Now everything is ready.

Let’s beging the training

After a loooooooooong time training the accuracy for the test model improved from 14.8% up to 37.7%. I’ve stopped because the rate of learning was very slow and improvement will take more time. Although the accuracy is not that great the coding proved that the model was learning (which was the goal afterall).

Hope it was somekind usefull and thanks for reading!

If anyone wants to use the code it is available on my profile at Kaggle

https://www.kaggle.com/kmaciver
Is your Numpy optimized for speed?
By: Ori Cohen
Tags: Numpy, Towards Data Science, Pandas, Python, Machine Learning

Is your Numpy optimized for speed?

The Numpy Logo, Wikipedia.

If your research work is highly dependent on Numpy-based calculations, such as vector or matrix additions and multiplications, etc. Then it is advisable to run a few checks in order to see if Numpy is using one of three libraries that are optimized for speed, in contrast to Numpy’s default version.

There are 4 Blas and Lapack flavors available and as far as I know, Numpy will grab one of the following (2,3,4) libraries and will default to the first one if neither exists in your system.

Libraries:

Default BLAS & Lapack OpenBLAS Intel MKL ATLAS

Benchmarks

Luckily, two benchmarks are available on the net, the first was made in 2015 and compares OpenBLAS, Intel MKL vs ATLAS, it tips the scale towards MKL and ATLAS, but it’s worth noting that a lot has changed since 2015.

Figure 1: a comparison of four BLAS libraries

The second one (Figure 1) is from 2017 and is more complete. It shows a comparison of for all four libraries and the results are in favor of OpenBLAS, Intel MKL vs ATLAS, while MKL takes the slight lead. The comparison was made for a vector and matrix multiplication, SVD, Cholesky decomposition, and eigendecomposition.

Which Library?

If you want to know which one Numpy is currently using, the short answer is :

import numpy

print(numpy.show_config())

The long answer is that you have to dig in the shell and see which one is installed.

For Numpy < 1.10:

ldd /<path_to_site-packages>/numpy/core/_dotblas.so

For Numpy≥1.10:

ldd /<path_to_site-packages>/numpy/core/multiarray.so

immediately after you will have to follow some symlinks.

$ldd /usr/lib/python2.7/dist-packages/numpy/core/_dotblas.so

linux-vdso.so.1 => (0x00007fff12db8000)

libblas.so.3 => /usr/lib/libblas.so.3 (0x00007fce7b028000)

libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fce7ac60000)

libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fce7a958000)

libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fce7a738000)

/lib64/ld-linux-x86-64.so.2 (0x00007fce7ca40000) $ readlink -e /usr/lib/libblas.so.3

/usr/lib/openblas-base/libblas.so.3

Please note that these examples were pasted from this SO post.

Many Thanks for reading this post, if you have a comment, feel free to reply.
Principal Component Analysis (PCA) using python (Scikit-learn)
By: Luca Chuang
Tags: Numpy, Pca, Sklearn, Python, Machine Learning

These are 3 steps for us to get the principal components

Standardizing the data. Obtaining the eigenvalues and eigenvectors of the covariance matrix. Sorting the eigenvalues and cumulate them.

Standardizing the data.

Eigendecomposition

We need to make the covariance matrix when we are doing eigendecomposition. For those who are not familiar with eigendecomposition, you can check this out: Eigendecomposition and PCA, this video explains the role of the covariance in eigendecomposition thoroughly.

A.T A is the covariance matrix

## Raw Eigenvalues:

## [6.17980179 1.48891782 1.24286411 0.85643648 0.80067244

## 0.62055361 0.53548571 0.40782165 0.07238151 0.26928531

## 0.21920788 0.16128337 0.18211551]

## Percentage of Variance Explained by Each Component:

## [0.47402652 0.1142086 0.09533486 0.06569363 0.0614162

## 0.04760005 0.04107485 0.03128228 0.00555208 0.02065574

## 0.01681451 0.01237137 0.01396931]

Because we have 13 variables in our X, we got 13 PCs.

Sorting the eigenvalues and cumulate them.

Sorting the eigenvalues by decreasing order to rank the eigenvectors and cumulate them to figure out how many PCs are we going to use.

## array([0.47402652, 0.58823512, 0.68356998, 0.74926361,

## 0.81067981, 0.85827986, 0.89935471, 0.93063699,

## 0.95129273, 0.96810724, 0.98207655, 0.99444792,

## 1.])

This plot looks exactly the same as what we learn PCA in JMP.
Solving D414: Section has no content
By: Nicole Carlson
Tags: Linting, Numpy, Docstring, Python

Today I went down a long rabbit hole trying to solve a linting error for a coworker. Our team at ShopRunner uses numpy docstrings, and we check our code with flake8 and flake8-docstrings.

My coworker had written a method that looked like this:

def my_sweet_method(self, examples): """

This is a dope docstring.



Parameters

----------

examples

str inputs to be tokenized Returns

-------

Tokenized strings

"""

...

When we ran flake8, we kept getting this error “D414 Section has no content” on the first line of the docstring (“This is a dope docstring”). This error was super confusing to us since the docstring clearly has content.

I spent about an hour trying to research this D414 error and found nothing useful online. My problem-solving method ended up being me deleting various parts of the docstring and running the linter over and over again.

Eventually, I realized that the error seemed to be related to the Parameters section of the docstring, which was clearly not blank. After a much longer time than it should have taken, I finally got the error to disappear when I replaced the real content under Parameters with random text.

Why did this fix the error?

Well, it turns out that flake8 does not look for sections via -------- ; instead it does keyword matching to find specific sections as designated by numpy. Examples is one of the special keywords that is often a section in numpy docstrings. Because we had named one of the variables examples and documented it in the docstring, the linter thought there were three sections: Parameters , Examples , and Returns .

So lesson learned: do not name variables after numpy sections.

For a full list of possible sections, see here: https://numpydoc.readthedocs.io/en/latest/format.html#sections. There are probably more than you think.
Tiny difference between pandas and numpy std()
By: Vladimir Belobragin
Tags: Numpy, Pandas, Python, Standard Deviation, R Language

Every time we get data we count standard deviation and average. And no one waits for a pitfall here — but it does for Python numpy library!

This is a nice example from the book “Applied Statistics with R” by David Dalplaz (you can find it at https://daviddalpiaz.github.io/appliedstats/) On page 55 two samples (sampled from normal distributions) for future t-test distribution calculation:

x = c(70, 82, 78, 74, 94, 82)

y = c(64, 72, 60, 76, 72, 80, 84, 68)

The book is illustrated by R code, and we obtain correct standard deviations for samples:

> sd(x)

[1] 8.294577

> sd(y)

[1] 8

But what if we try to calculate standard deviation in Python? Usually we use Numpy library, so let us try:

import numpy as np

X = np.array([70, 82, 78, 74, 94, 82])

Y = np.array([64, 72, 60, 76, 72, 80, 84, 68])

s_X = X.std()

s_Y = Y.std()

print(‘X std =’, s_X,’

Y std =’, s_Y)

it follows:

X std = 7.571877794400365

Y std = 7.483314773547883

This is not the same to what R sd() function provides. Now, try to use pandas std() method, and in this case the result is equal to R:

import pandas as pd

x = pd.DataFrame(X, columns = [‘series1’])

y = pd.DataFrame(Y, columns = [‘series2’])

s_x = x.series1.std()

s_y = y.series2.std()

print(‘X std =’, s_x,’

Y std =’, s_y)

X std = 8.294576541331088

Y std = 8.0

So if this is a contradiction or mistake? Sure not.

The trick is simple — the np.std() function has “ Delta Degrees of Freedom” (ddof) parameter, which by default is zero. And the same parameter is 1 for std() method in pandas (and 1 in R), what is theoretically correct when we work with samples.

To obtain exact numpy.std() figures, we set ddof parameter to 1:

s_X = X.std(ddof=1)

s_Y = Y.std(ddof=1)

print(‘X std =’, s_X,’

Y std =’, s_Y)

And voilla:

X std = 8.294576541331088

Y std = 8.0

So simple, so boring… but the mistake may be frustrating.
Higher Order Numeric Differential Equations(Python)
By: Rahul Tarak
Tags: Numpy, Differential Equations, Lorenz System, Python, Runge Kutta

Background

To provide some background as to why we care about truncation error, and why we need to use higher-order methods to achieve lower truncation errors

If you are confused about the Taylor Expansion, I’d recommend 3blue1brown’s video on the subject

What is truncation error?

From calculus, we know that the truncation error is the next term in the Taylor series, the (n+1)th term of the expansion. That is they are errors from approximate an infinite sum, as partial sums.

In terms of a computer, there is a term called machine epsilon or double precision. This is the accuracy beyond which a computer cannot tell the difference between two numbers. That is a truncation error of machine epsilon, while in mathematics is not considered an accurate result and still an approximation, in scientific computing it is considered accurate.

Machine Epsilon

In the above equation, h or dx is the time step. The equation has a truncation error of ~O(h), which means to decreases the error from e-8 to e-10, the new time step would have to e-2 smaller than the old-time step. Which makes the increase in accuracy for Euler’s Method extremely slow.
Generate matrices together with NumPy
By: Pavlo Horbonos
Tags: Numpy, Software Development, Programming, Data Science, Python

1. Sequences

You may be surprised, as I have stated the 2d-arrays as the topic of the article. Though array to matrix conversion is very wide-spread, so this makes sense. Let’s explore a pair of NumPy native tricks.

For example, for the sequence:

Reshape it:

Increase the dimensionality:

Since we have the instrument to convert an array to a matrix, we can apply it to a traditional np.arange() sequence generator, for example:

np.arange(1, 12, 2).reshape(2, 3)

2. Random numbers

Of course, we cannot pass by the random sequences, since they are overused in testing purposes. Numpy provides a massive number of different random distributions based on different formulas. Though we will review the most common ways to create random matrices, all other methods may be used similarly:

np.random.random([3,3]) — 3x3 matrix with the samples from the uniform [0,1) distribution

np.random.randn(3, 3) — 3x3 matrix with the samples from a univariate “normal” (Gaussian) distribution of mean 0 and variance 1

np.random.randint(0, 2, size=(3, 3)) — 3x3 matrix with random integers from the [0,2] interval.

3. Distributions

Again we work with the arrays, but we cannot miss such important functions, which produce continuous distributions of the data in the stated interval.

np.linspace() — linear distribution (with the equal step)

np. logspace() — logarithmic distribution

np.geomspace() — geometrical distribution (with the equal factor)

4. Array replication

It is a good example of a filler function, which takes an array and replicates it several times to get a matrix. We have to set the number of row and the number of repetitions in a rows (in that order).

5. Diagonal matrix

For the intermediate operations with matrices, we may need the diagonal ones. Also, it is one of the basic elements in linear algebra. Not a surprise, that NumPy provides several functions for their creation.

From array

Square matrix with ones

1s matrix of custom size with custom diagonal

6. np.zeros()

It creates a matrix completely filled with 0s. It is the perfect candidate for the default value of a variable or dumb return value.

You can also use np.zeros_like() to create a 0s matrix with the size of a custom one.

7. np.ones()

Another example of a default-valued matrix, filled with 1s for this time.

Again, you can use another array as a template with np.ones_like() .

8. np.full()

Previous functions create matrices with the defined filling values. But NumPy also provides a function for a custom filling.

9. Triangle matrix

Another important element of the linear algebra operations: matrix a half filled with 0s. NumPy has three main functions for triangle matrices:

np.tri() — creates a triangle 1s matrix with filled lower part

— creates a triangle 1s matrix with filled lower part np.tril() — creates a triangle matrix from a custom arry with filled lower part

— creates a triangle matrix from a custom arry with filled lower part np.triu() — creates a triangle matrix from a custom arry with filled upper part

10 Completly cusom matrix
Web scraping RSS feed using Python
By: Darshita Patel
Tags: Numpy, Beautifulsoup, Data Scraping, Data Science, Python

#100DaysOfCoding

Here we are going to compare RSS feed of different news channels of the same demographics and later we will compare both news channels' sentiment analysis.

Github: Click Here

RSS Feed is written in XML. So dealing with one or multiple RSS Feed should be the same. Here is an example of the BCC News RSS Feed.

Open Jupyter notebook and import beautifulsoup library.

Beautiful Soup is a library for pulling data out of HTML and XML files. It provides ways of navigating, searching, and modifying parse trees. More about Beautifulsoup: https://www.crummy.com/software/BeautifulSoup/

To install this package with conda run:

conda install -c anaconda beautifulsoup4

prettify() The function will turn a BeautifulSoup parse tree into a structured Unicode string, with a separate line for each tag and each string:

Beautiful Soup will allow a string match against any item in that list. This function will find all the <a> tags and all the <b> tags: soup.find_all(["a", "b"])

Now let’s find all the content embedded between item tags.

time to peek inside each item tag. It contains <title>, <link>, <discription>, <guide> etc.

Now let’s group all the tags separately and save them into an empty variable. Each item tag is saved from address 0.

Now to export the data into a csv file, first create a dataframe using pandas and save it into a CSV file.

The file contains the RSS news feed.

Code: https://github.com/DARSHITA188/RSSfeedParser_SentimentAnalysis

Move next: Sentiment Analysis of news feed using python

#100DaysOfCoding

#Day1
Remove green screen in python using numpy and opencv without any predefined functions.
Tags: Numpy, Opencv, Python, Image Processing

Remove green screen in python using numpy and opencv without any predefined functions. Nagarajramachandran Feb 8, 2020·2 min read

Welcome friends.

The above image consists of three images. The first image is key image. The second image is background image. The third image is composited image.

This code is not using any predefined functions. But the concept is the same. From the key image remove the area that contains the green values and replace it with background image.

The pseudo code is

The python code is

I took the 1080p image. The first step is to load two images. Then I loop over the height and width of the key image.The img[i][j][0] denotes blue,img[i][j][1] denotes green,img[i][j][2] denotes red. We want to check the pixel where the green colour occurs. So I put img[i][j][1]>150.The other two (blue and red) wants low values. So I put less than 30 and 50. This value is not constant. I play around with these values to get this output. But make sure of that in RGB green channel contains high values and other two channels contain low values in case that is a green screen.

The time taken is 17 seconds for a 1080p image. This is not an efficient output. I tried on my own without using any predefined function like threshold, masking etc. Thank you for reading this and love learning
Toxic!
By: Where Angels Fear
Tags: Humor, Politics, UK Politics, Brexit, Humour

Toxic!

I’m sure that freak/abomination SouthpawPoet will have something to say about my linking to The Guardian but, nevertheless, along with some of the articles being genuinely good (most notably those by John Crace and Marina Hyde), a lot of the Below The Line (BTL) comments are frequently superb (more consistently so than those on The Independent ¹ ) … not altogether infrequently some of the best writing you’ll see anywhere … and well worth the time taken to read through ².

Many of them on the below Politcs Sketch had me crying with laughter …

Although, I usually prefer Marina Hyde’s specifically political (rather than her Lost In Showbiz) articles, I laughed hard at the one below (one of her best) but then I read the comments section and all but soiled myself — I’d create an account and join in the fun but Life’s too short (so many retards, so little time 😉 )

And, although I felt the article itself was one of her weakest, this one has some good (funny) BTL comments as well …

Not to mention the inestimable Stewart Lee — who, it would appear, has been plagiarising me.

And a serious one …

The article itself is a bit lightweight … so so, not unself-congratulatory (dare I even say onanistic?) … but there are some superb comments BTL ³.
Eligibility for UK Spouse Visa or Marriage Visa UK and Fiancé Visa UK:
Tags: Uk Marriage Visa, Immigration, Uk Spouse Visa, UK Politics

Eligibility for UK Spouse Visa or Marriage Visa UK and Fiancé Visa UK: QuickInfoClub Nov 14, 2019·2 min read

Marriage visa UK or UK spouse visa and UK fiancé visa can be approved; if you will accomplish the compulsory conditions like:

1. The age of both of you must be above 18.

2. There must be no illegal relationships.

3. You both would have met each other personally.

4. Your relationship should be genuine.

5. The marriage must be done with the intention of a permanent relationship.

Financial Affairs:

To get UK marriage visa or UK spouse visa and UK fiancé visa; proofs of income are required for you and your spouse:

· Both of you have to prove the total income of 18600 pounds for the past 12 months. Moreover, applying for children requires further 3800 pounds for each child.

· The incomes can be earned through employment or self-employment too.

· There can be other specified sources of income.

· Your sponsor and you can apply by showing the savings of over six months maintained. They can be approved after deducting 16000 pounds from the aggregate amounts, then if the remaining savings would be summed up as 18600 pounds minimum after dividing them by 2.5.

· Specified pension income can be accepted too. Besides that, bereavement and maternity incomes in the UK are also applicable.