fabiochiu/medium-articles huggingface dataset:
Clustering Instability
By: Eva Van Rooijen
Tags: Cluster Analysis, Customer Segmentation, Unsupervised Learning, Clustering, Machine Learning

Clustering Instability

Clustering is an unsupervised learning technique used to create clusters of data points. An example is customer segmentation in marketing.

There are several clustering algorithms available. However, they require the number of clusters (k) to be given as input. Selecting the number of clusters can be difficult since this is an unsupervised problem, without labels.

Clustering Instability can be measured to determine the performance of the clustering algorithm, given k. We would prefer a cluster to be stable, but what does this mean?

Let’s say you have a sample of observations (X), we can shuffle this dataset and take three subsamples. These three subsamples are from the same sample X so they should follow the same structure.

We take our clustering algorithm of choice and train two separate versions of the model. First, a model is fitted on dataset z1. Second, a different model is fitted on dataset z2. In both cases, the same algorithm is applied given a specific value for k, the number of clusters.

model1.fit(z1) model2.fit(z2)

After both models have been trained, we predict the labels of the third (validation) subsample z3.

model1.predict(z3) model2.predict(z3)

To measure the instability of this cluster mapping, we calculate it as follows:

for every observation i and it's neighbour j = i+1

and it's neighbour check if model 1 agrees they have the same label: label1(xi) = label1(xj)

check if model 2agrees they have the same label: label2(xi) = label2(xj)

If model 1 labels both observation i and j equal, but model 2 on the other hand labels both different this causes instability of the cluster mapping.

If agree1 != agree2: instability += 1

This leads us to an instability score for this k and these subsamples. We want to repeat this process for a range of k to assess the clustering instability for all.

Finally, we select the number of clusters (k) which minimizes the instability.

Feel free to reply with any questions on the theory or how to code.

I am currently working on an application for selecting the number of topics in topic modeling with a Latent Dirichlet Allocation, let me know if you would like to see a follow-up!
Quantified Self: Clustering data from my Habit Tracker
By: Carlos Abiera
Tags: Habits, Routine, K Modes, Clustering, Quantified Self

I’ll be contributing to the Quantified Self movement by sharing the result of this clustering project.

The data exported from Loop — Habit Tracker. This is my friend for several months in my 8 to 5 job before. It helped me record the progress of my routine (what gets measured, gets done).

A habit is an action we do subconsciously or when there is no mental effort to it, otherwise it is a routine. These are the routine that I was trying to develop:

Morning Meditation, Morning readings, Midday Sleep, Out on time, Arthour (personal projects), Before sleep readings, Before sleep meditation.

Using the exported data from the app, I have different column and rows. Columns are the habit that I want to track, and rows are the binary values that show whether the routine was fulfilled or not (1-yes , 0-no) .

Raw Data

Bar Plot of fulfilled routine

Data Normalization

Pearson Correlation

Using correlation analysis, I discovered the strengths of relationships among my routines. This is worth exploring to establish a connection between these habits.

Interpreting Correlation Values

From the interpretation above we can see that:

Out on time routine (+0.63) has a moderate to strong positive linear relationship to Before sleep reading routine

Midday Sleep and Morning Readings routine (+0.73) have a strong positive linear relationship to each other.

Arthour and Before Sleep routine (+0.77) have a strong positive linear relationship to each other.

Before sleep meditation and Morning Meditation routine (+0.62) have a strong positive linear relationship to each other.

CLUSTERING

Clustering is an unsupervised learning algorithm that finds clusters or group in a set of data. These groups should have similar properties or features. This method is a common technique for statistical data analysis used in many fields.

K-modes Clustering. According to the documentation

K-modes is used for clustering categorical variables. It defines clusters based on the number of matching categories between data points.This is in contrast to the more well-known k-means algorithm, which clusters numerical data based on Euclidean distance.)

For simplicity, I used 3 number of clusters, this is a subjective judgement about the number of really distinctive clusters described here (I tried more K but it hurts my data).

These are different clusters derived using K-modes. I arbitrarily named my routines as: The Idealist, The Supervisor, The Visionary.

CLUSTER 1: The Idealist

This group has no Out on time and has a plenty of Midday Sleep routine. I remember I stay late at work to read or finish something that needed the next day.

CLUSTER 2: The Supervisor

This cluster has the least number of routines. These days could be tiring.

CLUSTER 3: The Visionary

This cluster has the most number of accomplished routines.

Analyzing such routines helps me understand better how my day looks like back then. From this data, I can design more meaningful routine towards a better day.

Originally published at https://www.linkedin.com.
RFM Clustering on Credit Card Customers
By: Dery Kurniawan
Tags: Data Analytics, Towards Data Science, Marketing Analytics, Clustering, Rfm Analytics

RFM Clustering on Credit Card Customers

Photo by Adeolu Eletu on Unsplash

Recency, Frequency, & Monetary (RFM) is one of the techniques that can be used for customer segmentation and is one of the conventional ways for segmentation that been used for a long time.

Recency refers to when the customer did the most recent transaction using our product Frequency refers to how often customers do transactions using our product Monetary Value refers to how much does a customer spend in our product

RFM method is straightforward; we only have to transform our data (usually in the shape of transactional data) into data frame consists with three variables Recent Transaction, Transaction Frequency, and Transaction Amount (Monetary Values).

Transactional data itself is the data which records or captures every transaction that been done by customers. Typically, transactional data consists of transaction time, transaction location, how much amount our customers spend, which merchant the deal took place, and every detail that can be recorded at the moment transactions were made.

Let see our transactional dataset that later will be used as our study case. Our dataset is a 2016 credit card transactional data from every customer. Transactions dataset consist of 24 features which recorded during every transaction our customers made. Even if we have many features on our dataset; we will not use all of them and only use small numbers of features which can be transformed into Recency, Frequency, and Monetary Values instead.

Link to dataset: https://www.kaggle.com/derykurniawan/credit-card-transaction

Fig-1. Transactional Data Features

If we return to our description of RFM features; we only have to keep customerId, transactionDate, and transactionAmount to create Recency, Frequency, and Transaction Amount features in the new data frame that grouped by customerId features.

For the Recency feature, we can subtract the current date with the maximum value of transactionDate (latest transaction). Since our dataset only contains 2016 transactional data, we will set 1st January 2017 as our current date.

For the Frequency feature, we count how many transactions were made for every customer using n() function in R.

for the Transaction Amount feature, we calculate the summation of transactionAmount for every customer.

Import and Transform Transactional Data to RFM Data

Fig-2. First six rows of RFM Dataset

Now we have three main feature for the RFM segmentation. It is similar to any other data analytical case, the first step that we have to do is exploring our dataset, and in this case, we will check every feature distribution using histogram plot using hist() function in R.
Data Science — Segmenting and Clustering Neighborhoods in Dubai and Doha
By: Rafael Cidade
Tags: Clustering, Data Science, Python, Cluster Analysis

1. Introduction

This article is the second part of my capstone project for Coursera IBM Data Science Professional Certificate, a 10-course program offered by IBM that explores several disciplines of the Data Science field. Some of those disciplines were applied to this project, which is based on Python.

1.1. Background

Dubai and Doha are cities located on the coast of the Persian Gulf, and they are very similar to each other. Not only because of their geographic location, climate and futuristic skyscrapers, but also for both being ranked among the most high tech and safest cities in the world.

Doha and Dubai are situated on the coast of the Arabian Peninsula (Google Maps)

Dubai is the most populous city in the United Arab Emirates (UAE) and the capital of the Emirate of Dubai. The population of Dubai is estimated at around 3,400,800 and its Economy represents a gross domestic product (GDP) of US$102.67 billion.

Doha is the capital and most populous city of Qatar. Its population is estimated at around 2,382,000. In terms of Economy, Doha’s GDP is around US$146.09 billion.

The life cost and average salary in both cities are similar too. According to the website Livingcost.org, the cost of living in Doha is 2% less expensive than in Dubai.

Doha vs Dubai comparison (https://livingcost.org)

1.2. Problem description

Dubai and Doha are also very similar in terms of opportunities for work, especially in IT. Based on that, we may face the following situation: an IT professional, based in Dubai, who got a great job offer from a big company in Doha, decides to move to Qatar’s capital.

In Doha, the professional would like not only to live near the new job but also to settle in somewhere similar to where he or she is currently based. Naturally, if we are used to live near places that make our lives easier and more confortable (such as gyms, restaurants, groceries), we’ll look for neighborhoods with the same characteristics when moving to a new city.

So, the challenge here is to use Data Science to help that professional in finding the best neighborhood.

2. Data

In this project, I basically used the following data: list of neighborhoods in Dubai and Doha (containing names and geolocation of each one of them), and Foursquare data (list of venues) about each neighborhood.

The source of the list of neighborhoods, in Doha and in Dubai, is Wikipedia. There are specific pages that list neighborhoods of both cities, with basic information, such as name and population, and a link to the neighborhood wiki page, where is informed its geolocation.

Foursquare is a technology company, and one of its products is Foursquare City Guide, an app that provides recommendations of places to go near a location. Foursquare’s API is the source of the main neighborhood data that I used in the project.

3. Methodology

Using Foursquare API and some Data Science tools and techniques, I segmented and compared neighborhoods of the two cities.
Categorical Data, Jaccard’s Coefficient, and Multiprocessing
By: Casey Whorton
Tags: Data Science, Clustering, Distance, Data Analysis, Categorical Data

Quick Example to Set the Stage

So, when comparing two sets (which can be an array, a series, or even a vector of binary values) the numerator is the count of elements shared between the sets and the denominator is the count of elements from both sets. In our case, the denominator is the size of the either set, so we can also say that this similarity score is the number of shared elements divided by the number of elements that could be shared.

Let’s check out a simple example:

from sklearn.metrics import jaccard_score

from scipy.spatial.distance import jaccard x = [[1,1,1],[1,0,1],[0,0,0]] print(x)

[[1, 1, 1], [1, 0, 1], [0, 0, 0]] jaccard(x[0],x[1])

0.33 jaccard_score(x[0],x[1])

0.66

The array x has three rows. The first row will be the observation we wish to compare to. Notice how the Jaccard function returns the number of elements not shared between the first two rows. The jaccard_score function returns the opposite: it’s the number of elements shared between the first two rows. One shows the dissimilarity and the other shows the similarity. I personally prefer the similarity score offered in scikit-learn, but it’s important you are aware of the difference.

(Further be aware that some people make the distinction that the element 0 shouldn’t be included at all in the calculation. This makes sense when under certain circumstances.)

Now that we’ve seen this metric under a simple case, let’s apply it to a larger dataset.
What is Machine Learning: Supervised, Unsupervised, Semi-Supervised and Reinforcement learning methods
By: Serafeim Loukas
Tags: Data Science, Artificial Intelligence, Clustering, Classification, Machine Learning

For this family of models, the research needs to have at hand a dataset with some observations and the labels/classes of the observations. For example, the observations could be images of animals and the labels the name of the animal (e.g. cat, dog etc).

These models learn from the labeled dataset and then are used to predict future events. For the training procedure, the input is a known training data set with its corresponding labels, and the learning algorithm produces an inferred function to finally make predictions about some new unseen observations that one can give to the model. The model is able to provide targets for any new input after sufficient training. The learning algorithm can also compare its output with the correct intended output (ground truth label) and find errors in order to modify itself accordingly (e.g. via back-propagation).

Supervised models can be further grouped into regression and classification cases:

Classification : A classification problem is when the output variable is a category e.g. “disease” / “no disease”.

: A classification problem is when the output variable is a category e.g. “disease” / “no disease”. Regression: A regression problem is when the output variable is a real continuous value e.g. stock price prediction

Some examples of models that belong to this family are the following: SVC, LDA, SVR, regression, random forests etc.

2.2 Unsupervised machine learning algorithms/methods
Coffee, Tacos and Data Science
By: Steve Atkin
Tags: Coursera, Clustering, Data Science

Applying Data Science

With all of the venue data in hand I would now be ready to apply some data science to the problem. The first task was to determine the type of analysis I would perform. I decided to cluster the venues to find similarities and differences in the venues close to the schools in San Antonio.

Preparing Data for Clustering

Clustering is a type of unsupervised learning. I decided to use the K-means clustering algorithm to cluster the venues. The K-Means algorithm is one of the most frequently used methods of unsupervised learning and typically yields good results. To learn more checkout this article on K-Means.

One challenge that I faced is that K-Means clustering requires numerical data as input. In my case I did not have numerical data, but rather categorical data. This meant that I would have to somehow map or transform the categorical data into numeric values. This is fortunately a common occurrence so there are good techniques for doing addressing this issue. The technique that I selected to use is called One Hot Encoding. This method maps the categorical venue data to numeric data that can then be used in our clustering algorithms. Checkout this article to learn more about One Hot Encoding.

This is a sample of what the data looks like once we Hot Encode it and then group the data by the name of each college, as shown below.

Venue Data Hot Encoded

Selecting Optimal Cluster Size

Before I could start the actual clustering I still had one additional issue and that is what is the optimal number of clusters I should use to segment my data? Once again this is a common problem in data science and many techniques exist to help answer this question. The technique I selected is called the Silhouette method. When you use the Silhouette method you select various cluster sizes and then score how well they actually cluster the data. Once you have this data you plot this on a graph and select the cluster size that yields the highest score. To learn more checkout this article on the Silhouette method. Here is the plot for my data and as you can see the optimal cluster size is 4, as shown below.

Silhouette Score

Clustering

With the data prepped for clustering and the optimal clustering size selected I could then feed the data into K-Means. Once the data was clustered I sorted the results by cluster value and displayed the top 5 venue categories, as shown below.
Support Vector Machines- An easy interpretation of categorizing inseparable data
By: Namrata Kapoor
Tags: Data Science, Support Vector Machine, Kernel Trick, Clustering, Machine Learning

In Machine learning the Support Vector Models are used for supervised learning based on associated learning algorithms for classification and Regression based analysis.

If you have examples of data marked as belonging to one of the classes, SVM training algorithm assigns new data one of the classes as non-probabilistic binary linear classifier.

An SVM model is a representation of the data points in space, mapped to each class so that the examples of the separate classes are divided by a clear gap that is as wide as possible. Examples of new data are then mapped into that same space and predicted to belong to a class based on the side of the gap on which they fall.

In addition to linear classification, SVM can also categorize non-linear data using a kernel -trick.

When supervised learning is not possible and the data is unlabeled, data clustering approach is taken to categorize the data into one of the category, this learning is unsupervised and applies support vector clustering algorithm.

This support vector algorithm is used to categorize the unlabeled data into its nearest category.

When the original problem is stated in finite dimension plane, the problem to categorize the data may not be possible in linear space.

It was thus proposed to have a higher dimension in which data can be separated and classified, to keep the computational load low so that that the pair of input vectors may be computed easily in space by defining in terms of Kernel Function.

Kernel Trick

Kernel trick is actually giving a solution in a plane which doesn’t exist originally in problem.

If we get a set of data as below we see that these cannot be categorized linearly with a straight line.

If we have the values as mentioned in image below of x-y axis, and plot them on 2D plane of x-y, we see that it cannot be categorized.

To add a new plane to it, we need to have a value for 3rd dimension which will easily separate out the red and green data point’s category.

We will try 3 equations here.

1) x + y

2) xy

3) x**2

As mentioned in the table above we can clearly see that xy formula has separated out red and green data points with values 0 and 2 respectively.

Now add 3rd dimension as z axis and plot the values of xy on it. We can see that these are easily separable with red data points going down and green going up.

To show this in linear space we know the solution is between xy=0 and xy=2 i.e. xy=1 , y=1/x, which is exactly the shape of line shown in image above.

It is to be noted that working in a higher-dimensional feature space increases the generalization error of support-vector machines, although given enough samples the algorithm still performs well.

Thanks for reading!
Clustering Budapest neighborhoods using location services data
By: Szakos Péter
Tags: Location Services, Jupyter Notebook, Clustering, Python, Machine Learning

Getting to know a city is not always straightforward, especially when we talk about country capitals with dozens of neighborhoods. Many times it can be useful to group similar neighborhoods, so one can have a better understanding what to expect when visiting certain parts of the city. In case of Budapest, there are more than 160 neighborhoods. It is a huge overhead to process and group them manually. On the other hand, it seems obvious to use location services data for the analysis, where people are already indicating what are the points of interests to them.

With this article, I would like to demonstrate what machine learning can tell us about Budapest.

I was using Jupyter Notbook/Python while processing/analysing the data.As a start, I extracted Budapest neighborhood names from a WikiPedia article. Additionally, I collected GPS data from the Nominatim database. Venues data was coming from the FourSquare location service API.

I put the Budapest neigbhorhoods on a Folium map:

Budapest neighborhoods on a Folium map

Using FourSquare data, I collected the venues for each neighborhood:

Venues dataframe

Afterwards, I created a pandas dataframe that contains the top 10 venues for each neighborhood:

Master dataframe — Budapest neighborhoods with the top 10 venue categories

I used this table as an input for k-means clustering. For the resulting clusters, I created a Folium map together with a WordCloud for the top venue categories.

All clusters on one map:

Cluster 0 — Hot spots:

Cluster 1 — Quiet spots:

Cluster 2 — Commuters’ outposts:

That’s it, thank you for reading this blogpost!

In case you are interested in the Jupyter/Python code in more detail, you can find it on GitHub, together with the interactive maps:
Instacart: A Not-So Insta-Analysis
By: Madison John
Tags: Predictive Modeling, Clustering, Instacart, Segmentation, Machine Learning

Model Preparation

Photo by Anna Shvets from Pexels

Dataset Inspection

Number of Unique Values for Each ID Variable

Above is a table showing the number of unique values for each of the identifier variables (aisle_id, department_id, user_id, etc.).

We will aggregate the data several times in the upcoming sections so we do not have to work with millions of observations. I am doing this work on my mid-tier home PC after all.

Dataset Transformations

Transforming From CSVs into Dataframes

The above graphic shows how information from each of the csv files was combined into three dataframes to better associate orders, users, and products ordered.

Additionally, the products ordered were grouped into the departments from which they came and aggregated as sums. This reduced the observation count from ~32 million to ~3.4 million.

Finally, a new variable num_items was added to record the total number of items ordered per order.

Data Visualization

Now let’s take a look at some pretty pictures to learn a little bit about the data.

Top 10 Aisles by Number of Products

Above are the ten aisles with the most number of products. Good to see vitamins and supplements round out the top three after candy, chocolate, and ice cream.

Top 10 Departments by Number of Products

Above shows the ten departments with the most number of products.

Top 10 Products Ordered

Above are the ten products ordered most often. Fruits and vegetables are quite popular with Instacart users.

Top 10 Departments by Products Ordered

Now we have the ten departments with the most number of products ordered. As alluded to in the previous graphic and confirmed here, produce items are the most ordered on Instacart.

Now let us start looking at the orders with respect to time.

Number of Orders by Day of Week

The above graphic shows that during the weekends, specifically on Saturdays and Sundays, are when the greatest number of orders are placed. The number of orders decreases toward the middle of the workweek, with Wednesday having the least number of orders.

Number of Orders by Hour of Day

The plot above indicates there is a peak ordering period between 10 AM and 4 PM, with the number of orders decreasing gradually toward midnight and a dead period following that until 6 AM.

Number of Days Elapsed Since Previous Order

Looking at the order lag time, or the number of days elapsed since the previous order, we can note several groups:

monthly shoppers

shoppers every ~3 weeks

shoppers every ~2 weeks

weekly shoppers

shoppers every 3–4 days

Number of Orders by Order Number

Above, we see that all users in the dataset placed at least four orders. After that, there is a marked decline.

Number of Orders by Number of Items Ordered/Re-Ordered

Finally, we have the distribution of the number of items ordered and re-ordered. Note the long tails to the right from the peaks of five and two items ordered and re-ordered respectively.
K-means Clustering on Ordinal Data
By: Ariel Wentworth
Tags: Data Science, Unsupervised Learning, Clustering, Deep Learning, Machine Learning

K-means Clustering on Ordinal Data

Image by Reimund Bertrams from Pixabay

K-means clustering. It’s the holy grail of unsupervised learning. And honestly? I understand why… Sure, there’s a bit of an art form to deciding on the number of clusters you should calculate, but by and large it’s borderline magical to sit back and let the algorithm do it’s thing. Even so, there’s one very important caveat: k-means clustering only works on numerical data… Right?!

Well… Maybe

In general, attempting to broaden k-means into categorical applications is precarious at best. The most integral part of k-means clustering deals with finding points with the minimal distance between them. How do we define distance amongst categorical variables? How far is an apple from an orange? Are those closer to blueberries or watermelon? In unordered contexts, it just doesn’t make sense to ask these questions.

But what if my categorical data is ordered?

Wow, I’m so glad you asked. Just as you’ve astutely pointed out, there absolutely are contexts where it seems like there should be a way to answer these questions.

Recently, I was looking at some data from KFF regarding the statewide social distancing mandates that have been implemented in the United States in response to COVID-19. After some wrangling, here’s what the first few rows looked like:

(Note: If you want to see my whole process for wrangling and analyzing this dataset, you can check out my R code on Github.)

Consider for instance, the Stay at Home Order attribute. This attribute describes whether or not a state has a Stay at Home order, and which populations are affected by the order. Obviously having no order is different from having a statewide order, which is different from having an order that only applies to high risk groups. And here’s the important part: a statewide order seems “closer” to an order affecting only high risk groups, than no order at all.

It seems like we should be able to answer the questions required for k-means clustering. The natural ordering of our data lends itself to further analysis. Sure, it’s not actually numeric data, but it behaves in a roughly numeric way, so what’s holding us back from treating it as such? In the ubiquitous words of Hannah Montana, life’s what you make it so let’s make it possible to perform k-means clustering on ordinal data! (No? That wasn’t her quote? Shoot.)
Week 6 — Clustering. In data mining, this will be one of the…
By: Curiosity To Data Analytics, Career Journey
Tags: Data Science Marketer, Data Science, Dendrogram, Clustering, K Means

In data mining, this will be one of the method of data mining. Data mining tools’ capabilities and ease of use are essential (Web, Parallel processing, etc). Beside we learned about association, prediction, time-series relationship, we will now learn more about cluster, which is part of segmentation.

Different type of data

Type of clustering:

K-means (PartitionING method)

Basically, this is assuming that a fixed number of clusters, ‘k’ and create as much as compact clusters. What I understand from k-means algorithm, we can get different numbers from randomly initialize clusters and assign data points to nearest clusters.

But, how many clusters could we create?

We can use elbow method. The idea of the elbow method is to run k- means clustering on the data set where ‘k’ is the number of clusters

Within the sum of squares (WSS), it is defined as the sum of the squared distance between each member of the cluster and its centroid.

After studying the behavior of a population, you have identified four specific individual types that are valuable to your study. You would like to find all users who are most similar to each individual type. This study will be be the most appropriate for this algorithm.

2. Hierarchical (Agglomerative method — Dendrogram, Tree, sub-clusters)

It is divided into 2: Divisive (top-down) and Agglomerative (Bottom-up)

Meaning all items is in one cluster or the other way around. Each item is in its own cluster.

The Graph is generally known as Elbow Curve.

• Red circled a point in above graph i.e. Number of Cluster =6 is the point after which you don’t see

any decrement in WSS.

• This point is known as the bending point and taken as K in K — Means.

This is the widely used approach but few data scientists also use Hierarchical clustering first to create dendrograms and identify the distinct groups from there.3. Nearest neighbor clustering

Discrete random variables

Discrete and continuous random variablesDensity-based

in R, it is an unsupervised learning, groups are not pre-define.

Hard vs fuzzy (soft) vs biclustering

Distance or similarity measures

Distance = inverse of similarity

Gender is symmetric attributes:

Distance between Jack and Mary

Distance (similarity) threshold

Graph representation

Studies on machine learning algorithms can be used for inputting missing values of both categorical and continuous variables. K-means clustering, linear regression, K-nearest neighbor, and decision trees.
Improving Collaborative Filtering With Clustering
By: Jackson Wu
Tags: Recommender Systems, Clustering, Data Science, Machine Learning

Now, let’s go into greater detail on how k-means fits in with collaborative filtering and how k-means needs to be modified to work with sparse input.

K-Means

First, a brief conceptual overview of k-means.

K-means works by finding the most central point for each of the k clusters, choosing the closest data point to that center, re-constructing new clusters based on those centroids, and repeating those steps until convergence.

In user-based collaborative filtering, these sample data points would have n dimensions since each user is represented as a row in the m x n matrix. The process of forming the clusters looks like this:

For all rows u of the m x n matrix (users), we assign u to the closest centroid using a distance function (like the Manhattan distance and the Euclidean distance). Reassign each of the k centroids to the centers of their newly formed clusters. Repeat.

Dealing With Sparsity

The main way we accommodate the sparse input is to only use specified ratings. We can find the mean of each cluster by only using the specified ratings within that clusters. We can also find the distance between each user and each centroid by only using the common specified ratings between each pair.

(Note that all of the above works for item-based collaborative filtering when we cluster columns instead of rows.)

So now, say we have implemented this correctly, and we now have k clusters of users. Now, every time we have to compute pairwise similarity for some user, we only compute this similarity across users in the same cluster!

This will drastically improve runtime, especially when the cluster are fine-grained. Keep in mind, this efficiency, as always, comes at a cost — this time, its accuracy.

Prediction function (from Recommender Systems: The Textbook)

From here, we’re golden. Once we have the peer group, it’s the same as before. We can use the same neighborhood prediction function from the user-based collaborative filtering we’ve already seen, and voila: piping hot predicted ratings.

Clustering isn’t the only way to efficiently find peer groups for neighborhood-based methods. You can also improve the training stage of collaborative filtering with dimensionality reduction, which I cover in this article.
Multi-Level Cluster Promises More Effective Systems
By: Ant Media Server
Tags: Clustering, WebRTC, Media Server, Streaming Server

What is a Multi-Level Cluster?

Ant Media Server provides highly scalable solutions with a cluster structure. We have a blog post and documentation about cluster structure. In this blog post, we will introduce a new feature which is the multi-level cluster.

A cluster that has different regions is called Multi-Level Cluster where each region has its own origin for a stream. Ant Media Server can be scaled in different physical locations when you create CDN like a cluster. In this structure, you can distribute the nodes to regions based on their physical location. This has some advantages in performance and data-transfer. Because each region will have its own origin node for a stream, the edges in a region will pull the stream from the origin of their region.

How does it work?

Let’s clarify the case with an example scenario. We have two scenarios

Publisher & Players in the same cluster Publisher & Players in the different clusters

Publisher & Players in the same cluster

Publisher starts a stream and it is assigned to the Origin1 instance in the Region1 Player1 who is geographically closed to Region1 instance requests to play the stream. Player1 is assigned to Edge11 instance in the Region1 Since the Origin1 is the origin of the stream, Edge11 will pull stream from Origin1 .

Publisher & Players in the different clusters

Player2 who is geographically closed to Region2 requests to play the stream. Player2 is assigned to Edge21 instance in Region2 Edge21 checks the origin of the stream.

Since there is no local origin instance in Region2 to pull the stream. It assigns itself as the secondary origin for that stream in Region2 .

to pull the stream. It assigns itself as the secondary origin for that stream in . Edge21 pulls the stream from the Origin1 (main origin) in Region1 .

pulls the stream from the (main origin) in . Edge21 sends the stream to Player2 .

4. Player3 who is geographically closed to Region2 requests to play the stream.

5. Player3 is assigned to Edge22 in Region2 .

6. Edge22 checks the origin of the stream. Since Edge21 is a secondary (local) origin for the stream in the Region2 , it pulls the stream from Edge21 and serves to Player3 .

In short, each stream is distributed from the main node that was the closest node to the publisher. Also, all regions will have their own secondary (local) origin for a stream if someone close to that region requests to play the stream.

How to configure Multi-Level Cluster?

Region for a node is determined with the nodeGroup parameter. You can set the node group (or region) of a server by adding nodeGroup=GROUP_NAME in conf/red5.properties file. Make sure that the instances running in the same region should have the same nodeGroup value.

Also, this parameter can be set on the fly with the REST method.

Keep in mind that you should configure the Load Balancer to forward Publish and Play requests to the best region. If you have a global cluster in different geolocations, you should have a Load Balancer like AWS Route53 Service.
Clustering method 6
By: Yuki Liu
Tags: Machine Learning, Clustering, Cluster Analysis, Unsupervised Learning

－ 算法實例 －

import numpy as np

import matplotlib.pyplot as plt

from sklearn.cluster import OPTICS

from sklearn import datasets #create datasets

X,y = datasets.make_blobs(n_samples=50, centers=3, n_features=2, random_state= 20, cluster_std = 1.5) #parameter setting

eps = 2

MinPts = 5 #OPTICS method

model = OPTICS(MinPts, eps=eps, cluster_method='dbscan')

#Use cluster_method='dbscan' to compare with DBSCAN method

model.fit(X)

labels = model.fit_predict(X) #results visualization

plt.figure()

plt.scatter(X[:,0], X[:,1], c = labels)

plt.axis('equal')

plt.title('Prediction') plt.show()

（標示顏色僅是分群結果，沒有順序性）右圖為 epsilon=2 的結果

plt.figure(figsize = (15, 10))

plt.axis('equal') # plot all data

plt.scatter(X[:,0], X[:,1], c = labels) # rearrange the order of data

X_all = X[order,:]

seq_labels = labels[model.ordering_] for classes in range(3):

class_idx = seq_labels == classes

class_X = X_all[class_idx,:]

plt.plot(class_X[:,0], class_X[:,1]) for i in range(class_idx.sum()):

point = class_X[i,:]

plt.text(point[0], point[1], str(i)) plt.title('Ordering')

plt.xlabel('x')

plt.ylabel('y')

此圖為同一群中資料點的排序，紫色點則為雜訊點
A Quick Guide on PDF Data-Cleaning in Google Sheets
By: Wambugu Gichuki
Tags: Data Science, Google Sheets, Excel, Data, Data Cleaning

Working with PDF data can be a pain, especially when having to deal with tables that are in PDF format.

In today’s world of cross-format data analysis, one finds themselves having to deal with data of different formats and needing to change them to different formats to make them usable for analysis as a pandas data-frame.

Recently I started a project to visualize in detail the 2019 Kenya Population and Housing Reports and unfortunately for me, all the data was in PDF format including the all important data tables. Alas my journey began with taking a deep dive into PDF data cleaning techniques and eventually culminating with having to learn google sheets formulas.

The problems faced in successive order include:-

Extracting the pages of interest from the reports (Data tables)

Converting the extracted PDF’s into excel format

Cleaning up the extracted tables and formatting them properly for analysis

After plenty of research (nerdspeak for “I just googled it all”) I was able to find some 2 useful tools for accomplishing the first 2 tasks.

For extracting the pages from the reports I suggest using Split PDF files online. Free service to split PDF — Ilovepdf which I found to be extremely useful considering that most importantly it is completely free! Yes there are actually paid versions of this stuff that are way worse at doing the job!

After extracting our pages of interest, we need to convert them to excel format and after many attempts with multiple types of software for this job (including some more popular options in the python ecosystem) which all failed miserably at the task, I decided to go with PDF Converter To Excel — Convert from PDF to Excel online. Not only does it render clearer outputs than the closest copycats out there but it also has an API available and supports clear multipage document conversion. The key to this is that it uses OCR (Optical Character Recognition) which is honestly far better and clearer than anything out there right now. Plus its a free tool so no fuss about having to pay up for one time usage.

After this the tables extracted vary in quality but most data is captured well enough.
Django Admin Export to Excel, CSV, and Others
By: Adi Ramadhan
Tags: Export, Excel, Django

Export data in django admin into excel spreadsheet, csv and other format using django-import-export library. Official docs here.

Step 1: Preparation, Create Django Project, Initial Migration

create virtualenv: virtualenv venv

start virtualenv: venv/Scripts/activate

install Django in virtualenv: pip install django==3.0 django-import-export

Create Django: django-admin startproject myproject

Go to myproject folder: cd myproject

Initial Migration: python manage.py migrate

Step 2: Create Django Apps

Create apps: python manage.py startapp myapp

Add myapp to INSTALLED_APPS in myproject/settings.py

Step 3: Create Model in myapp/models.py

Step 4: Register Model in myapp/admin.py and Add ExportActionMixin

Step 5: Makemigrations and Migrate

Make migrations: python manage.py makemigrations

Migrate: python manage.py migrate

Step 6: Create Superuser

Create superuser: python manage.py createsuperuser

Type username, email password and retype password

Step 7: Run Server and Testing

Run Server: python manage.py runserver

Access: http://127.0.0.1:8000/admin

Add some data, select data, choose format and go (export).
Working with Spreadsheets in Flutter: Part 2
Tags: Mobile, Flutter, Excel, Storage, 30daysofflutter

Shall we begin

As we did in the previous session, I believe we have added excel package and Imported it to our main.dart file

loading an existing document from storage

To load an existing file from storage we need the help of file_picker plugin

file_picker: ^2.1.6

You might need to stop and rerun your application so you don’t get methodChannel Exception.

declare a field called selectedExcel;

2. Paste the following code into your app dont worry about the getList() we will create it in a minute you can comment it out for now

here we pick an excel document and assign in to the variable “selectedExcel” that we create earlier

3. Create a FlatButton in your app and make onPressed call the pickFile() method

FlatButton( color: Colors.blue, onPressed: pickFile, child: Text( "Pick from storage", style: TextStyle(color: Colors.white), ))

4. Prepare an excel document. for this tutorial this is the document I’ll be using

It contains First name, Last name and profession. you can format your excel document how ever you please.

5. click on the flat button you created earlier, grant permission and select your desired excel file.

6. If you have no errors then you have successfully picked your excel file from storage.

Reading data from an excel file

This is where we create the getList() method we talked about earlier.

Remember in our last session we said excel files can have many sheets and by default the first sheet is called “Sheet1”.

Here we loop through the first sheet inside the excel that we selected and print out its values to console.

If you did everything right then your console should display your data like this

Congratulations you have successfully read data from your excel file. As you can see each row is a List.

Passing the rows into a List

All we have to do now is to create a list of rows, don’t forget that each row is also a list

Create a List variable called “tbleRows” Uncomment the “tbleRows.add(row)” inside the getList() method

If you followed through with the above steps you should have a list containing all the rows on your selectedExcel file

Creating a card to display the data from your list

Column( children: [ Expanded( child: ListView.builder( padding: EdgeInsets.all(0), physics: BouncingScrollPhysics(), itemCount: tbleRows.length, itemBuilder: (BuildContext context, int index) { return ListTile( title:

Text("${tbleRows[index][0]} ${tbleRows[index][1]}"), subtitle: Text(tbleRows[index][2]), ); }), ), ], )

Incase you are confused about “tbleRows[index][0]”: tbleRows is a List of List; where each nested list (Inner List) is a row in the excel file;

I hope I haven't lost you?

And there you go, we have successfully fetch and displayed the content of our Excel file.
Best Office Document Reader App For Android
By: Office Suite
Tags: Pdf Reader, Document, Document Reader, Excel, Office 365

Office Document Reader, file viewer allow you to view and read all document files on your mobile.

Docx reader and viewer is the smallest size and all-in-one complete free office suite app.

✪ Document Manager

The office viewer allows you to manage and arrange all document files in folder structure view.

All document files are also available in a single place which is very easy to search and view.

✪ File Viewer

Document viewer / Document reader for android lets you easily view Word, Excel, PowerPoint, Text and PDF files. It also supports multiple compatibility with office formats including DOC, DOCX, XLS, TXT, XLS, PPT, PPTX and PDF.

✪ PDF Creator / PDF Editor / PDF Converter

PDF converter option allows you to convert file from PDF to word converter, PDF to jpg converter, PDF to doc converter. The Image to Pdf Converter (jpg to PDF, png to PDF) easily groups and converts your images into a single PDF file. Cropping tool allow you to scaling & optimize your images and also can create PDF files from user input text

✪ PDF Viewer / PDF Reader

Read PDF files easily just tap tap and done.

Fast and stable performance

PDF File View allow you to Zoom-in and Zoom-out for perfect vision

Search, create, save the pdf file quickly

Easily share & send PDF Files

✪ XLSX Viewer — Excel Reader

Xls viewer read all excel file formats

View file xls, xlsx with high quality view

Xlsx File Reader or xls file viewer does not required any internet connection to view

✪ PPT Reader / View PPTX Slide

Easily browse & open powerpoint and slides, presentations file files on device

✪ Doc Viewer / Doc Reader

Docx Reader or docx viewer is the best & fast way of reading Word documents on your mobile phone. Word viewer is a simple & lightweight app. Docx file readers represent all formats of documents in the best way

✪ Folder Structure

List of all file in folder view structure

✪ Quickly Search

Quickly open any Word, PowerPoint, Excel, Text, and PDF by using search option

✪ File Information

Directly file open and easy to pick and see file information like file path, file size, last modified date etc and easily share documents.
#Juststart Project
By: Shariq Hassan
Tags: Excel

For a few years now, I’ve wanted to master MS Excel, the spreadsheet program by Microsoft. In fact, I have wanted to be good at it from the very beginning of my mandatory training at an accounts firm that started back in mid 2015. Understandably, its been quite a while. Most business professionals have at least some basic familiarity with the program and while I’ve learned some of the basics, the potential to improve is endless. As an accounts and finance professional, it is both an essential requirement and an additional skill that sets you apart from your peers.

The first three tasks I need to do in order to master MS Excel are (i) to identify the sources of learning, (ii) to gather the data (in whatever form from these sources) and most importantly, (iii) set aside time everyday to actually study, practice and learn new features.

I have already gone through with the first two steps - or at least, partially. I have listed down the people I will have to track to gather the lectures, presentations and raw dis-aggregated data for practice on Excel and the online courses I will have to consult.

I have corresponded with an acquaintance who is a certified Excel trainer and I’ve had him email me his material. I have also signed up for a three part course by Microsoft itself on edX that deals with excel and data analysis. Now its upto me to set apart the time to actually wind up the planning phase and start the operational phase.

It is the 12th of January, 2019 today and Ii am hoping to making everyday count.
Select Non-Adjacent Cells
By: Beanstalk Cfo Group
Tags: Excel Tips, Excel

By selecting multiple cells in Excel you can delete data, apply formatting such as borders or shading, or apply other options to large areas of a worksheet all at one time.

While using the drag select method with the mouse to quickly highlight a block of adjacent cells is probably the most common way of selecting more than one cell, there are times when the cells you want to highlight are not located beside each other.

When this occurs, it is possible to select non-adjacent cells. Although selecting non-adjacent cells can be done solely with the keyboard, it is easier to do using the keyboard and mouse together.

Selecting Non-Adjacent Cells in Excel

For help with this example, see the image above.

1. Click on the first cell you want to select with the mouse pointer to make it the active cell.

2. Release the mouse button.

3. Press and hold down the Ctrl key on the keyboard.

4. Click on the rest of the cells you want to select them Without releasing the Ctrl key.

5. Once all the desired cells are selected, release the Ctrl key.

6. Do Not click anywhere else with the mouse pointer once you release the Ctrl key or you will clear the highlight from the selected cells.

7. If you release the Ctrl key too soon and wish to highlight more cells, simply press and hold down the Ctrl key again and then click on the additional cell(s).