fabiochiu/medium-articles huggingface dataset:
Rape As A Political Weapon- Looking Back At CPI(M)’s Worst Legacy In West Bengal
By: Pratyush Adhikary
Tags: West Bengal, Rape Culture, Left, Communist Party Of India

So normalized was rape as a political weapon that even after the Left Front was defeated, rape threats were (and are) still commonly used without the batting of an eyelid. The national media shook when Tapas Pal, one of the most celebrated actors in Bengali cinema and an MP from Krishnanagar was caught on cam threatening rape of CPI(M) women, but for most of the residents of the state, it was almost something routine.

Tapas Paul’s rape threat an old political tool in Bengal

The legacy of the Left is being carried forward by the Trinamool Congress.

https://indianexpress.com/article/india/politics/tapas-pauls-rape-threat-an-old-political-tool-in-bengal/

When Tapas Paul threatened his political rivals that he would get their women raped, the Trinamool Congress MP was only resorting to a well-entrenched political culture in West Bengal — to use rape and the threat of rape as a tool for vendetta. Largely a legacy of when the Left was in power and now being carried forward by the ruling Trinamool Congress’s supporters, political enmity has often degenerated into the worst forms of criminal acts against women. …during the Left’s long stint in power, several of her party cadres were accused of such crimes. In fact, it was political infighting within the Left in Nadia district that resulted in the infamous “Dhantala rapes” in 2003. A marriage party bus was waylaid, at least six women were gangraped and the driver of the bus was killed. The incident happened when the Left was in power and investigations later showed that it was the result of an internal conflict between two senior CPM leaders in the area. The family of one of the CPM leaders was supposed to have been on the bus after a marriage reception late in the night. But the goons attacked the wrong bus while their target bus arrived a little later.

Just imagine this. You want to settle scores with a political contender and the best way you decide to do so is to rape his or her family members?

Or as a commoner you’re taking a routine bus journey until you’re stopped at the gunpoint, gang-raped by people with red flags covering their faces only to learn later that they “attacked the wrong bus”.

Another incident from the times when the Left front was steadily gaining prominence was the infamous Ashok Kumar night, where women have dragged out, raped, murdered and dropped in a pond, but was justified by Jyoti Basu who went on to rule West Bengal for 23 years as the CM as “rise of the proletariat”. Here’s the CPI mouthpiece “Frontier” deflecting on it.
Our Children Are Watching
By: The Office Of Superintendent Of Public Instruction
Tags: Government, Civic Engagement, Education, K 12 Education, Civics

Below is Superintendent Chris Reykdal’s statement on yesterday’s events in Olympia and in Washington D.C.

Editor’s Note: This story was originally published in Chris Reykdal’s Medium publication, which was sunset in May 2021. Statements from Superintendent Reykdal will continue to appear in the main OSPI feed.

Superintendent Chris Reykdal

OLYMPIA — January 7, 2021 — Yesterday, we watched with disgust and sadness as both our nation and state capitol buildings were bombarded by groups of our own citizens attempting to undermine the will of the people in a democratic election.

This was an overt attack on our government and our democracy, and it was fueled by hate and by a complete disregard for some of the key pillars of our society.

We have a responsibility to recognize how protestors in previous situations were treated so differently by law enforcement than the rioters yesterday. The privilege — the privilege based on race — that was on display yesterday was astounding.

Although we have made strides, we remain a nation founded on racism, classism, and inequality, and those prejudices are still embedded within our society and our institutions today.

As a former history teacher and as a parent of two public school students, I know our young people are watching these events unfold and they are looking to us for contextualization. Educators and families are answering questions and providing reassurance to our students — even as they are making sense of the events themselves.

Part of our state’s mission for education is to prepare our students for effective, collaborative, and civil civic engagement. Yesterday’s events make abundantly clear that our schools must engage and empower students, from an early age, with opportunities to participate in civil conversations, examples of effective civic engagement, and tools to find peaceful solutions to community problems.

This moment must transcend politics. We must join together to restore faith in our democracy and continue to model the ethical and civil behavior that we want to see from our own children.

Our young people are watching.

For More Information

· E Pluribus Unum: Resources for Talking to Students About the Attack on the U.S. Capitol
Thanks Tom, your useful exercise takes the “dosage” metaphor to a logical next step.
By: Jonathan Fox
Tags: Accountability, Civic Engagement, Governance, Participation

Thanks Tom, your useful exercise takes the “dosage” metaphor to a logical next step. This framework both spotlights the puzzling, persistent practice of expecting just a few meetings to change the world — and reveals the term’s limitations.

My original intent with using that medicalized imagery was to communicate why low dose (“underpowered”) external interventions may be insufficient to encourage enough agency to give teeth to voice. https://bit.ly/2zqay71

Yet a framework focused only on an intervention’s ‘intensity’ addresses just the first links in the long causal chain on the path to the goal of accountability. While the intensity of external support certainly matters, whether and how those ‘doses’ translate into agency is a different question.

This is where the dosage metaphor hits the wall. It risks implicitly treating the targets as passive, homogeneous recipients, rather than conscious actors who are likely to cautiously assess whether ostensible external allies can really be counted on — before sticking their necks out to speak truth to power.

Who takes what kinds of action, and why? Who are the allies, especially within the state, and what do they do? Who resists accountability, and how? Do external allies proactively reduce risks of collective action, and if so, how? These key theory of change questions should frame theories of action (interventions).

So yes, variation in dosage of external enabling actions for participation matters a lot. So does variation in breadth, depth and intensity of collective action — not to mention variation in resistance to accountability. Iterative cycles count too.

We need dynamic models — plus evaluation methods that do not render invisible the likely variation in patterns of action — and can identify positive outliers even when the needle seems not to move (if one only looks at averages).

PS: The ACT Health intervention discussed in your blog was one element of a broader change initiative. After the RCT stopped, the action continued with more intensive support to people-centered advocacy campaigns. @participangela and @VincentMujune are writing up broader lessons. Watch this space: https://accountabilityresearch.org/publications/
Welcoming new team members to the American Journalism Project
By: Jason Alcorn
Tags: Civic Engagement, Local News, Democracy, Nonprofit News, Philanthropy

Our all-hands meeting this morning

Over the last year we’ve announced our first grants, started working closely with our grantees — and just last week introduced our first CEO, Sarabeth Berman, who is stepping into the big shoes that John Thornton created as our day-to-day leader for the last 14 months.

What’s exciting is the enthusiasm we see around local, nonprofit news. What’s somewhat daunting is the scale of the work, and the new urgency that the current national crisis has forced upon us.

At the same time, we’re buoyed by the response of the field and of philanthropy. 44 states have made journalism an essential service. Many community response funds include relief and support for local news organizations. National journalism funders and platforms like Facebook and Google have committed additional funding.

We’re launching into our next phase of work now and have a great team in place. In fact, we haven’t stopped to introduce them, and wanted to make sure we did.

These four new colleagues bring to AJP experience that closely aligns with our values and mission to grow and sustain nonprofit, local news organizations and build a nationwide movement for local news. Please join us in welcoming them!
5 Important Ways To Participate In Civic Engagement
By: Rajkumar Surisetti
Tags: Rajkumar Surisetti, Civic Engagement, Philanthropy

Civic engagement is more popular than ever as people across all ideologies push for change. Anyone who cares about civic matters needs to take action to advocate for the causes they believe in. These five activities should be on anyone’s list.

Stay Informed

The first and most important step to civic engagement is to remain informed. Only get information from legitimate sources, and follow a number of them to be exposed to multiple points of view. Always thoroughly research every candidate and issue to form a fact-based opinion before voting, advocating or engaging in debate with others. Individuals will be far more able to present nuanced, detailed arguments as a result.

Advocate

Advocating can take many forms, including participation in protests, contacting elected officials, writing op-eds and more. Writing posts on social media and complaining to a like-minded bubble can only go so far. Action must be taken to drive real change, and those who are passionate and visible are the face of it.

Volunteer

Volunteering has always been and will always be the backbone of civic engagement. This is true whether cleaning up garbage in a community, working at a food pantry, registering voters or helping a non-profit get their message out. People should get out and support the causes they believe in to improve their communities and make the world a better place.

Engage

As difficult as it can be to engage with others who do not share a particular view, it is the only way to change minds and grow ideologically as a person. Learning about and discussing issues with others is one of the most effective ways to participate in civic engagement. Focus on common goals and shared viewpoints and then expand from there.

Vote

Finally, be sure to vote. Voting is the single most powerful thing anyone can do to affect real change in their communities. Don’t just vote in large, national elections but small, local elections as well. Every vote really make a difference at all levels of government. Vote for candidates as well as ballot measures.

Many of these ideas tie into one another. Individuals must be well-informed in order to advocate effectively, engage with other viewpoints and make smart voting decisions. Participate in civic engagement in a variety of ways to experience the best results.
Andre Lefebvre said
By: Miss Daisy
Tags: Covid 19, Civic Engagement, Social Justice, Humanitarian, Covert 1dollar

Andre Lefebvre said

…we have been sleeping at the wheel, making decisions based on ideas of prosperity rather than values of humanity.

As Americans, our big idea of prosperity has been the Grandiose American Dream: everyone can prosper if they look out for themselves; if they don’t, it’s their problem.

I picture Covid-19 as an experimental vaccine against ‘COVERT-$1’, a chronic condition that has ravaged the most vulnerable segments of our body politic for years. Symptoms include the social effects of Covid-19: unemployment, fear, food insecurity, unresponsive government, inaccessible healthcare, and unnecessary deaths…as a lifestyle.

I agree with Craig Allen Heath; we can all help.

Containing COVERT-$1 calls for widespread testing. The test is widely available, possibly in a mirror near you. (Some people will experience temporary pain.) Expect positive results if you were surprised by the ‘news’ that minorities and seniors are disproportionately impacted by Covid-19, or that a high percentage of underpaid ‘essential’ workers are women.

People with acute symptoms of COVERT-$1 will require several health measures including a public awareness campaign alerting them to the contagious nature of the disease, its overlapping symptoms, its origins and cross-generational perpetuation.

Recovery requires proven empowerment programs — treatments antithetical to the Tuskegee syphilis experiment, redlining, and similar COVERT-$1 infested projects.

Asymptomatic carriers of COVERT-$1 can make important contributions. Unlike COVID-19, the cure for COVERT-$1 begins with social alliancing. Purposeful association among people of differing backgrounds is the stimulus needed for a new, sustainable ‘normal’.

We gather together to dream and scheme, then take it from there.

As a ‘vaccine’, Covid-19 has dreadful side effects; but, as we have seen these past few months, abominable viral afflictions require severe countermeasures.

The metaphor stops here. No one ‘administered’ Covid-19. But we can — we must — take advantage of the hope hidden within this crisis. This may be the beginning of a new era of awakened, active and humanitarian citizenship. If it leads to a healthier nation, then those on the front lines and those who have succumbed to it have not suffered and died in vain.
Citizens Utilize SeeClickFix to Help Combat COVID-19
By: Mckay Burley
Tags: Civictech, Covid 19, Seeclickfix, Civic Engagement, Community

As new cases of COVID-19 continue to develop across North America, citizens are receiving communication from their city and taking action through SeeClickFix. Correspondence through SeeClickFix keeps citizens up-to-date as new information arises. Given the uncertainty we all feel, community leaders are taking steps in places where they know they can make a difference: stopping the spread and flattening the curve.

To flatten the curve and help serve at-risk populations, cities have created COVID-19 related categories such as: Help a Neighbor, Price Gouging/Business Closure, and Concerns & Questions. These categories help give power back to citizens who may be feeling helpless, allowing them to express their concerns or take impactful actions to help those who are most vulnerable.

Burlington, VT has created 21 categories, users can report under, to help with COVID-19 assistance related to their community’s health and wellbeing, either provided by the City or volunteers.

Now more than ever is an important time for citizens to participate and be connected with their communities. During these troubling times it’s best to lend a socially distanced helping hand to your neighbors and let them know “we are in this together”. Many of us are facing similar challenges as our day to day activities have been disrupted, but together we can work to build a stronger and healthier community for everyone.
Civic irresponsibility in Burkina Faso: example of the anarchical construction of speed bumps
By: Achille Sawadogo
Tags: Burkina Faso, Civic Engagement, Road Safety, Infrastructure

Speed bump made of clay

In popular jargon in Burkina Faso, they are called in french “gendarmes couchés” which literal translation is ‘lying police officers’. These are speed bumps, these bumps on the road to force users, especially motorists to slow down. They make it possible to limit accidents and save human lives when they are implemented in the state of the art. However, they can be very dangerous and lethal when the people themselves take the initiative to install them

In Burkina, Decree №2012–1111 of 31/12/2012 issued by the Council of Ministers sets the design and construction standards for speed bumps. It makes an inventory of the different types of speed bumps and establishes the construction methods. The speed bumps market is offered to professionals who do the work in the state of the art. However, non-standard speed bumps sprout like mushrooms on roads in Burkina Faso. They are most often encountered on unpaved roads. They are illegally installed by people who do not take into account the normal dimensions of construction or who do not take care to install a sign to warn users. Consequences: frequent road accidents with the result of many loss of life. Even if the Burkinabè government had begun in May 2018, an operation of demolition of speed bumps built anarchically on asphalted roads or not, it is clear that the continuous populations to erect them.

“Speed ​​bumps to reduce dust”: is this a valid reason?

Speed bump in a city entrance

For some inhabitants of Ouagadougou, the installation of these speed bumps acts as a breath of fresh air. “I live in Guikofê [neighborhood of the city of Ouagadougou]. There is too much dust; people are spinning with their gear. Yet the roads pass people’s courts. If there were no speed bumps, we would not be able to keep up. The speed bumps that the government sets up are only located on the national roads. We cannot do otherwise”.

The speed of users combined with the unpaved road structure can only cause a permanent state of dust. On the other hand, installing speed bumps anarchically is also the cause of many accidents. It is a dilemma situation that only the government can solve. It’s up to them to deploy their technicians in the cities of Burkina to set up quality speed bumps to the delight of the inhabitants, while securing the lives of users. Otherwise, equipped with their shovels and picks, the populations will continue to do as they wish, to the great displeasure of road users…
Can you teach “civic engagement” — and how?
By: Leaders Of Color
Tags: Civic Engagement, Community Organizing

Written by Dr. Lacretia Carroll, Training Director, Leaders of Color Memphis

Leaders of Color fellows study principles and applications of civic engagement in our proprietary curriculum

Spoiler Alert: The answer to this question is a resounding “yes, and we’re doing it every day.”

Academically, civic engagement is defined as “working to make a difference in the civic life of our communities and developing the combination of knowledge, skills, values, and motivation to make that difference. It means promoting the quality of life in a community, through both political and non-political processes.” (Excerpted from Civic Responsibility and Higher Education, edited by Thomas Ehrlich, published by Oryx Press, 2000, Preface, page vi.)

For our team at Leaders of Color, this definition of civic engagement is expanded to specifically include the work of Black and Latino community members to achieve racial, educational, and economic equity through intentional community organizing and civic action. To achieve this, we built our curriculum to help our fellows understand what it means to be civic leaders and offer them the knowledge, network, and skills needed to build their leadership skills and the future they want to see in their communities.

Leveraging evidence-based research from academia, Leaders of Color measures fellows and builds curriculum across all six dimensions of civic engagement as defined by the Association of American Colleges & Universities (AAC&U):

Diversity of Communities and Cultures

Analysis of Knowledge

Civic Identity and Commitment

Civic Communication

Civic Action and Reflection

Civic Contexts/Structures

Here are some examples of how our curriculum helps build strong leaders of color who are committed to advancing equity and building power in their communities:

The Leaders of Color program interviews current fellows and alumni annually to measure civic engagement activities and actions. We also do this to make sure that our program resonates and is relevant to our fellows and alumni. Together, we are building the future we want to see.
What California do you want? The future lies in your vote.
By: Lili Gangas
Tags: Polling Places, California, Civic Engagement, Recall Elections, Oakland

In less than 24 hours, on September 14, 2021, Californians will decide the future of our state.

Californians will vote on whether or not to recall Governor Gavin Newsom. The Governor has a lot of power to influence the direction of California. They can fight for or roll back progress through policies, the state budget, and the direction of state agencies and departments. Their words and leadership can change the course of who we protect and who we hold accountable for decades. The September 14 recall election is our opportunity to vote for our communities and make our voices heard about what we want to see for the future of California.

However, low voter turnout threatens the future of California.

This is why Kapor Center’s West Oakland site has been activated as a polling site from Saturday Sept 11th, 2021 through Tuesday Sept 14th, 20221. Our polling site is located at: 1901 Polar Street, Oakland, CA 94607. Our site is 1 of only 3 polling sites in West Oakland so our footprint is important.

And there is still time to make sure your vote counts and voices heard on September 14 before 10pm:

Go out & VOTE! Learn more about the recall, and what it means for your community. Click here to retweet our Kapor Center polling site, 1901 Polar Street, Oakland, CA 94607. If you are in West Oakland join us at 1901 Poplar to vote! Mobilize your family, friends, and community to turn out for the election before 10pm on September 14th.

Through community outreach and partnerships locally with KMEL 106, The Town Experience, artists and activists Ryan Nicole & Kev Choice, Red Bay Coffee, Oakland African American Chamber, Endeavors, and Agave Uptown we aimed to create a safe and family-friendly space for voters on Sunday and Tuesday. See the community in action here!

There were also grassroots organizations across California that we supported through grants. Learn more about their work below:

Our power is our vote, and our vote will decide the future of health, education, jobs, and equity in California. So you get to decide what California you want! Let’s #SMASHTheVote together!
Community Resilience Through the Civic Commons, Part 2
By: Reimagining The Civic Commons
Tags: Cities, Sustainability, Civic Engagement, Resilience, Community

​​Compared to the city as a whole, North Minneapolis is more racially diverse, has more young people, and — at an estimated $40,000 — has a lower average household income. It’s a proud community, with many assets made up of people, places and a culture that have been impacted by present and historic inequities, including redlining. “In North Minneapolis, we’ve talked — and we’ll need to continue to talk — at length about how we connect Northsiders to the river,” said Minneapolis City Councilor Jeremiah Ellison at the Overlook’s opening. “A big part of it is not only changing the landscape, but that it’s youth and folks on the Northside who contribute to what that landscape looks like. I’m excited to see how this will be a catalyst for access to the river.”

The Overlook elevates Black design within and for a largely Black community. James Garrett of 4RM+ULA Architecture was the lead architect of the Overlook, which features an 840 square foot oval loop trail and deck perched on the river’s edge. The deck is circumscribed by a steel railing which includes a 48- foot interpretive panel, designed by Samuel Ero-Phillips with contributions from Black youth designers at Juxtaposition Arts (JXTA). The panel depicts a timeline of the Mississippi Riverfront, from pre-settlement to its imagined future.

Minneapolis-based TENxTEN Landscape Architecture and Urbanism led the Overlook’s multidisciplinary design team. In a nod to the deep roots that nourished this project, James Garrett co-founded the Environmental Design Studio at JXTA (with Satoko Muratake of TENxTEN) and years ago, Councilmember Ellison was one of JXTA’s first youth apprentices.
Using Logistic Regression to detect if a feature is Numerical or Categorical
By: Yann Avok
Tags: Feature Engineering, Python, Data Science, Logistic Regression, Features

You read the title and you might wonder why someone would want to automatically detect if a given feature is either numerical or categorical. This will come in handy for a greater task because it is just a step to handle missing values in a dataset(for a general machine learning problem). The end goal is to be able to replace every empty cell with an ensemble method(Random Forest for example) by taking into account every variable type. But first, let’s focus on how we can actually achieve that aim.

What characteristics can we use to make a distinction between a numerical and a categorical variable?

In the subsequent sections, we will discuss and dissect every chunk of code that has been written in Python for the occasion. The full code is available here.

Categorical vs Numerical variable

A categorical variable is one that can have multiple classes or modalities(with or without order) whereas a numerical variable measures some quantity on every individual in a population.

Examples of categorical variables:

Hungry : Yes, No, Yes, Yes, No

: Yes, No, Yes, Yes, No Gender : 1, 0, 1, 1, 0, 1

: 1, 0, 1, 1, 0, 1 Pain: 1, 5, 2, 7, 1, 10

Examples of numerical variables:

Age : 15, 30, 26, 74, 25, 12

: 15, 30, 26, 74, 25, 12 Salary ($): 5000, 1000, 3000, 25000, 1500, 7500

Unique Values, Floating Numbers and Sequentiality

A categorical variable tends to have less unique values than a numerical one. It also presents no floating numbers compared to what we can observe in a numerical one.

We will consider a feature to be categorical if:

it has very few unique values (but not only)

it contains a character or a string

it contains integers (but not only)

We will consider a feature to be numerical if:

it has a significant amount of unique values

it contains floating numbers

it contains integers (but not only)

When a categorical variable is comprised of integers, the unique values are naturally sequential. A numerical variable will contain non sequential modalities.

Architecture of the repository

Repository

We have two main modules:

In data_type_identifier.py , we wrote a class for preprocessing the data, building our model and our prediction method.

, we wrote a class for preprocessing the data, building our model and our prediction method. In run.py we instantiate our class and execute everything.

At each step, we will be showing what has been written in the class DataTypeIdentifer and its execution in run.py.

Data Preprocessing

The following dataset has been entirely made up for the circumstance. Let’s have a look at it.
Categorical Variable Encoding Techniques
By: Shubham Singh
Tags: Data Science, Feature Engineering, Python3, Categorical Variable

A categorical variable is one that has two or more categories (values). There are two types of categorical variable, nominal and ordinal. A nominal variable has no intrinsic ordering to its categories. For example, gender is a categorical variable having two categories (male and female) with no intrinsic ordering to the categories. An ordinal variable has a clear ordering.

Many ML algorithms are unable to operate on categorical or label data directly. However, Decision tree can directly learn from such data. Hence, they require all input variables and output variables to be numeric. This means that categorical data must be converted to a numerical form.

Few types of categorical variable encoding are:

One hot encoding: Encoding each categorical variable with different Boolean variables (also called dummy variables) which take values 0 or 1, indicating if a category is present in an observation.

One-Hot Encoding

2. Integer Encoding / Label Encoding: Replace the categories by a number from 1 to n (or 0 to n-1, depending the implementation), where n is the number of distinct categories of the variable.

3. Count or frequency encoding: Replace the categories by the count of the observations that show that category in the dataset. Similarly, we can replace the category by the frequency -or percentage- of observations in the dataset. That is, if 10 of our 100 observations show the color blue, we would replace blue by 10 if doing count encoding, or by 0.1 if replacing by the frequency.

4. Ordered Integer Encoding: Categories are replaced by integer 1 to k, where k is the distinct categories in variable, but this numbering is decided by mean of target of each category. In example shown below,Green has target mean of 0, Red has target mean of 0.5, yellow has target mean of 1. Hence yellow is replaced by 1, Red by 2 and Green by 3.

5. Encoding using “Weight of Evidence”: Each category will be replaced by natural log of [p(1)/p(0)], where p(1) is the probability of good target variable and p(0) is the probability of bad target variable of each category in categorical variable. In famous “titanic” dataset, one of the categorical variables, “Cabin” can encoded as shown below, given that “Survived” as target variable. p(1) is the probability of surviving for each category and p(0) is the probability of death. Note: WoE is well suited for Logistic Regression, because the Logit transformation is simply the log of the odds, i.e., ln(P(Goods)/P(Bads)).
Hands-on with Feature Engineering Techniques: Broad Introduction
By: Younes Charfaoui
Tags: Feature Engineering, Python, Data Science, Artificial Intelligence, Machine Learning

Why Feature Engineering?

At the start of every machine learning project, raw data will inevitably be messy and unsuitable for training a model. The first step is always data exploration and cleaning, which involves changing data types and removing or imputing missing values.

With a certain understanding of the data acquired through exploration, though, we can solve these challenges and build high-performing models.

This may include removing outliers or specific features you don’t want the model to learn, as well as creating features from the data that represent the underlying problem better.

However, it’s important to understand that predictive analytics is not magic, and although an algorithm learns on a very basic level, it can only extract meaning from the data you give it. Algorithms don’t have the wealth of intuition that humans have (whether that’s good or bad), and consequently, the success of an algorithm can often hinge on how you engineer the input features.

Feature Engineering vs. Feature Selection

Often, newcomers to the field of machine learning may get confused between feature selection and feature engineering.

Feature engineering allows us to create new features from the ones we already have in order to help the machine learning model make more effective and accurate predictions.

Feature selection, on the other hand, allows us to select features from the feature pool (including any newly-engineered ones) that will help machine learning models make predictions on target variables more efficiently.

In a typical ML pipeline, we perform feature selection after completing feature engineering.

Here is another series about feature selection:
Understand Feature Engineering for Machine Learning in 5 minutes
By: Florian Bouron
Tags: Feature Engineering, Data Science, Artificial Intelligence, Algorithms, Machine Learning

How to do feature engineering?

Let’s see different strategies of feature engineering. In this article, we won’t see all the methods, but the most popular ones.

Adding and dropping features:

Let’s assume we do have the following features:

Price of houses

If we want to predict the price of a flat, the number of plants might be irrelevant. In that case, we need to remove this feature from our machine learning model to don’t add extra noise.

This noise is called the curse of dimensionality. This means that as the number of features in the data increases, the number of data points required to build a good model grows exponentially.

We need to choose which features have are the most relevant to our model.

Combining multiple features into one feature:

Price of houses

In the example above, we can see that square meters and square feet are actually the same data but not the same unit. If we give this to our algorithm, it will have to understand that the square meter and square foot are related and are actually the same feature.

That’s why we need to decide on which measurement to take and keep only one.

We could also have two features, number of dogs and number of dogs, and combine them under the number of animals.

Number of animals

Though, combining the features is not every time a good idea. For example, in the case of a date feature, probably the day of the week matter.

You need to remember that quality is better than quantity.

Cleaning existing features:

You need to keep the features that you think are relevant for your model picking up on the right signal in the data.

To do that you can:

Impute missing values.

Remove outliers for not trying to train with data points that are not representative.

Getting rid of the scales, for example, if you have features in centimeters and some other ones in meters, try to convert all of them in centimeters. This is called normalization .

. Transform skewed data to make it more compact for our model thanks to an easier distribution.

Binning:

Binning is when you take a numerical measurement and convert it into a category.

Here is an example for home sales:

Home sales

In that example, we can assume that the sale price depends on the fact that there is a swimming pool.

We can then simplify our model by pre-processing the data and replacing the swimming pool length with a boolean future.

Swimming Pool boolean

One-hot encoding:

One-hot encoding is a way to represent categorical data in a way that the machine learning algorithm will understand.

Our model understands numbers but not strings, that’s why we need to convert strings to numbers. Though, we cannot assign random numbers to our strings, because our model might give more importance to big numbers than little numbers. That’s why we are going to use a one-hot encoding.

Here is an example about home sales:

One-hot encoding

One-hot encoding is useful for replacing categorical data with simple numeric data that the machine learning model will understand.
Fast Feature Engineering in Python: Image Data
By: Sayar Banerjee
Tags: Feature Engineering, Computer Vision, AI, Data Science, Machine Learning

Libraries

import random

from PIL import Image

import cv2

import numpy as np

from matplotlib import pyplot as plt

import json

import albumentations as A

import torch

import torchvision.models as models

import torchvision.transforms as transforms

import torch.nn as nn

from tqdm import tqdm_notebook

from torch.utils.data import DataLoader

from torchvision.datasets import CIFAR10

Resize/Scale Images

Resizing is the most fundamental transformation done by deep learning practitioners in the field. The primary reason for doing this is to ensure that the input received by our deep learning system is consistent.

Another reason for resizing is to reduce the number of parameters in the model. Smaller dimensions signify a smaller neural network and hence, saves us the time and computation power required to train our model.

What about the loss of information?

Some information is indeed lost when you resize down from a larger image. However, depending on your task, you can choose how much information you’re willing to sacrifice for training time and compute resources.

For example, an object detection task will require you to maintain the image's aspect ratio since the goal is to detect the exact position of objects.

In contrast, an image classification task may require you to resize all images down to a specified size (224 x 224 is a good rule of thumb).

After resizing our image looks like this:

Why perform image scaling?

Similar to tabular data, scaling images for classification tasks can help our deep learning model's learning rate to converge to the minima better.

Scaling ensures that a particular dimension does not dominate others. I found a fantastic answer on StackExchange regarding this. You can read it here.

One type of feature scaling is the process of standardizing our pixel values. We do this by subtracting the mean of each channel from its pixel value and then divide it via standard deviation.

This is a popular choice of feature engineering when training models for classification tasks.

Note: Like resizing, one may not want to do image scaling when performing object detection and image generation tasks.

The example code above demonstrates the process of scaling an image via standardization. There are other forms of scaling such as centering and normalization.
Creating Clustering Feature importance
By: Guy Louzon
Tags: Cluster Analysis, Feature Engineering, Feature Importance, Clustering

I was thinking about using machine learning to create a musical editor. I found a great dataset on kaggle.com, started to play around with it, and design the process. During my attempts, trying to use K-means clustering, with various conditional samplings, I stumbled upon a question — how do I measure feature importance in clustering.

The need for it is simple: I would like to juggle between some feature to get a few various clustering on one hand, and to have netter clustering on another. The basic thing was to do, of course, was to google it, and since it’s a Dev question, prefer answers on stackoverflow.com.

Surprisingly, the search results and answers didn’t point to a simple enough solution that can be implemented, and I found myself trying to resolve it alone, so his is what I came up with:

Let’s say we have a dataset with two dimensions: x and y, and let’s also say that we have 1,000 records in this dataset: 500 records with the values of x=100, y=100 and 500 records with the values of x=900, y=900.

We obviously have 2 clear clusters, very distinguished. The reason is that the variance between any record, in each cluster, is zero, which makes them ideal

If we were to take this dataset, specifically the 500 records with the values of x=100 and y=100, keep their y values at 100, but change their x values, so that it would spread around a little bit, we will not necessarily be able to say that we have two clusters anymore… meaning that the larger the variance, the lesser the importance of the feature!

before we continue, small termination fix, for calculation purposes I preferred standard deviation, so I will use that from now on instead of variance.

The last issue to be handled is the scale. An average of 100, with a standard deviation of 50, might be a lot for the feature, but so far we’ve created a super simple example. it really is feature depandant: If the feature’s normal scale if between -500 and 500 we’ll treat it differently then if the scale will be between 0 and 150.

For that reason, we need to have the scale normalized. what we can do is take the maximum and minimum values for this particular feature, calculate their difference, and get their scale. Now, we can check the standard deviation in relation to the scale and get a normalized feature importance

Our feature importance, as it stands now, has two features: it’s between 0 and 1, and the lower the number, the higher the importance

Since our intuition is usually the opposite — the higher the number the more important the feature, we can simply normalize it again having it deducted from 1

so our simple formula for feature importance is:

1 — (std(x) / (max(x) — min(x))

and now, I can go about continuing to break my head with the ML musical editor

the feature importance class code is maintained here

more data related articles on the data channel

https://newsscroller.midrehov.com/42/6x7.php?v=webapp&channel=156
RecSys’10: Performance of recommender algorithms on top-n recommendation tasks
By: Arthur Lee
Tags: Paper, Machine Learning, Recommendation System

Recommendation system paper challenge (2/50)

Paper Link

Why I write this blog?

The main reason is that I want to take some notes so that in the future, I can quickly recall it and do have to add so many favorite papers for that. Besides that, this paper is really great and popular!

Why this paper?

RecSys’10 and it has high citation.

Furthermore, it do huge contribution to the current research. At that time, most researchers applied RMSE to recommend items to users.

However, they found improvements in RMSE often do not translate into accuracy improvements. Moreover, they provide better evaluation methods and better models.

The data?

Movielens, Netflix

The metric?

Recall and precision

What problem they want to solve? Recommending top-k problem

They applied Recall and Precision to measure the performance, which directly connects to the real on-line application.

What model do they propose?

1. Non-normalized Cosine Neighborhood

The original neighboring method is this formula (CorNgbr):

They proposed this formula (NNCosNgbr):

They remove denominator, which normalizes to rating range [1,5].

First reason is rating number does not matter for ranking.

Second reason is d controls the confidence, giving more weights for more number of reviewing. In this way, the estimated of raring is more reliable.

Besides, they don’t use Pearson similarity (only compute overlap items). They consider cosine-similarity , taking missing value as 0.

2. PureSVD

Traditional SVD:

How to deal with missing rating?

Traditional approach is estimating them as baseline. -> Too huge data, not feasible.

Later, some researchers consider ignore them and regularization to avoid overfitting.

As a result, there are two baseline model:

AsySVD: It represents users as a combination of item features.

SVD++: It does not represent users as a combination of item features. Yet, it is with highest quality in RMSE-optimized factorization methods.

Their model:(PureSVD)

Hence, r is observed, Q is the item features, qi is the target item vector. These values we can easily compute and get the score.

Here, r is score instead of a real rating.

Result
What is Location-based Recommendation?
By: Roshan Dwivedi
Tags: Alie, Recommendations, AI, Recommendation System, Business

A location-based recommendation engine analyzes the data related to the user location along with other related data to generate more accurate recommendations.

Nowadays, the majority of the online platforms (apps and websites) are adopting location-based recommender systems for the customers. Food delivery, salon services, gaming, and hospitality are some of the online businesses that use such recommendation systems to generate location-based suggestions for the customers.

The location-based recommendation systems are utilizing the growing usage of smart mobiles to collect the location-related data of the users.

Use Cases of Location-Based Recommendation Engine

Some of the use cases of the location-based recommendation engine are-

Location Based Social Networks (LBSN)- An LBSN adds a location to an existing social network. This enables the people in that social network to share their locations as the embedded information. Foursquare, Gowalla, Swarm etc. are some examples of such networks which suggest nearby tourist spots, next place to visit etc.

An LBSN adds a location to an existing social network. This enables the people in that social network to share their locations as the embedded information. Foursquare, Gowalla, Swarm etc. are some examples of such networks which suggest nearby tourist spots, next place to visit etc. Online Social Networks- Nowadays social networking sites such as Facebook, Twitter, and others use location-based recommendation systems to suggest local events, posts, trends etc.

Nowadays social networking sites such as Facebook, Twitter, and others use location-based recommendation systems to suggest local events, posts, trends etc. Online News Portals- Online news portals often use location-based recommendations to provide the relevant local news to the customers.

Location-based recommendation engines are getting increasingly popular among online platforms and it is not just limited to the above use cases but OTT channels, e-commerce stores, e-learning portals, and many more!

Muvi Recommendation Engine Alie provides real-time instant notifications based on the customer data. Alie is scalable, equipped with advanced algorithms and industry-agnostic.

Take a 14-day free trial to know about Alie.
Build a Movie Recommendation Engine frontend using Vue.js (Part 4)
By: Nikola Kuzmic
Tags: Movies, Vuejs, Recommendation System, Front End Development

This is the final part of our 4-part series! In the previous 3, we covered the theory of collaborative filtering, how to build a Flask API, and how to deploy the API on AWS ECS. Here are the previous posts:

In this post, we’ll build a simple Vue.js frontend that aims to simplify the movie recommendations as much as possible. Hence, we’ll only ask the user to enter their favourite movie and recommend movies that are similar to it.

Key functionalities of this project are the auto-search function for finding movie titles from MovieLens dataset and leveraging open-source scrapped movie posters to display recommended movies by the backend API.

Advice if you’re new to Frontend Development

In case you’re brand new to Vue.js (like me), I recommend first going through the fantastic Khan Academy resources which focus on HTML/CSS, JavaScript, DOM, and jQuery to familiarize yourself with the core frontend concepts.

Vue.js is one of the latest JavaScript frameworks which enables us to build interactive features in 10 vs 100 lines of code. In this article, we’ll be using Vue 2. The most important Vue foundational concepts you need to understand are:

i) Vue project structure

if you run a simple command

vue create vue_proj_ex

It will create an entire project that can be tested out using

npm run serve

The key aspects of the created template project to focus on are the following:

The main application setup is in the App.vue in which we load components such as HelloWorld.vue as well as any json data sub-directories. In main.js we load necessary libraries such as Bootstrap to help us leverage existing frontend style functionalities.

To do anything useful, we will have to declare variables and functions inside components such as HellowWorld.vue as follows:

ii) Declaring data methods (ie. variables)

data() {

return{

varX: '',

varY: [] } }

iii) Declaring functions

methods: {

fcnName(){ <Some Code>

} }

iv) One- and Two-way Data binding

For example, wherever you see

v-bind: or shortly :

is an example of a one-way bind. This means that the variable values stored inside your data method will be sent to the main app component.

Two-way binding

Is imposed through

v-model

this additionally enables variable values entered inside the app components such as forms to be transmitted and stored inside the data method variables.

Once you feel comfortable with JavaScript and DOM concepts such as events (ie. things that happen to HTML elements), you can move on to building out simple projects using Vue by leveraging excellent tutorials like this one. That will make it easier for you to understand what we‘ll do in the next section.

Let’s dive in!