Data Cleaning With Python and pandas
By: Christina Levengood
Tags: Numpy, Programming, Pandas, Python, Data Cleaning

Cleaning Text Data With NumPy where()

Next, let’s look at the violation_type column.

In [12]: df['violation_type'].head(10)

Out[12]:

0 E-ELEVATOR

1 C-CONSTRUCTION

2 E-ELEVATOR

3 E-ELEVATOR

4 LL1081-LOCAL LAW 10/81 - ELEVATOR

5 E-ELEVATOR

6 LL6291-LOCAL LAW 62/91 - BOILERS

7 ACC1-(OTHER BLDGS TYPES) - ELEVATOR AFFIRMATIO...

8 E-ELEVATOR

9 E-ELEVATOR

Name: violation_type, dtype: object

Call unique() on the violation_type column to see the unique values.

In [13]: df['violation_type'].unique()

Out[13]:

array(['E-ELEVATOR', 'C-CONSTRUCTION',

'LL1081-LOCAL LAW 10/81 - ELEVATOR',

'LL6291-LOCAL LAW 62/91 - BOILERS',

'ACC1-(OTHER BLDGS TYPES) - ELEVATOR AFFIRMATION OF CORRECTION',

'LBLVIO-LOW PRESSURE BOILER', 'AEUHAZ1-FAIL TO CERTIFY CLASS 1',

'EVCAT1-ELEVATOR ANNUAL INSPECTION / TEST',

'FISPNRF-NO REPORT AND / OR LATE FILING (FACADE)',

'JVCAT5-RESIDENTIAL ELEVATOR PERIODIC INSPECTION/TEST',

'VCAT1-ELEVATOR ANNUAL INSPECTION / TEST',

'FISP-FACADE SAFETY PROGRAM', 'HBLVIO-HIGH PRESSURE BOILER',

'LL2604S-SPRINKLER', 'LANDMK-LANDMARK', 'UB-UNSAFE BUILDINGS',

'PA-PUBLIC ASSEMBLY', 'B-BOILER', 'LANDMRK-LANDMARK',

'EVCAT5-NON-RESIDENTIAL ELEVATOR PERIODIC INSPECTION/TEST',

'RWNRF-RETAINING WALL', 'LL1198-LOCAL LAW 11/98 - FACADE',

'ES-ELECTRIC SIGNS', 'LL1080-LOCAL LAW 10/80 - FACADE',

'P-PLUMBING', 'BENCH-FAILURE TO BENCHMARK',

'LL11/98-LOCAL LAW 11/98 - FACADE',

'L1198-LOCAL LAW 11/98 - FACADE', 'EARCX-FAILURE TO SUBMIT EER',

'JVIOS-PRIVATE RESIDENTIAL ELEVATOR', 'CMQ-MARQUEE',

'LL10/80-LOCAL LAW 10/80 - FACADE', 'LL5-LOCAL LAW 5/73',

'IMEGNCY-IMMEDIATE EMERGENCY', 'LL2604E-EMERGENCY POWER',

'Z-ZONING', 'EGNCY-EMERGENCY',

'HVIOS-NYCHA ELEV ANNUAL INSPECTION/TEST',

'ACH1-(NYCHA) - ELEVATOR AFFIRMATION OF CORRECTION',

'ACJ1-(PRIVATE RESIDENCE) - ELEVATOR AFFIRMATION OF CORRECTION',

'CS-SITE SAFETY', 'LL10/81-LOCAL LAW 10/81 - ELEVATOR',

'LL2604-PHOTOLUMINESCENT', 'A-SEU', 'CLOS-PADLOCK',

'LL16-LOCAL LAW 16/84 - ELEVATOR',

'HVCAT5-NYCHA ELEVATOR PERIODIC INSPECTION/TEST'], dtype=object)

It’s not clear exactly what some of these are, but many of them are related to elevators, boilers, construction violations, plumbing, and that sort of thing.

Let’s pretend that I just want to look at the most common types of violations in general.

We can look at the value counts for each.

In [27]: df['violation_type'].value_counts()

Out[27]:

E-ELEVATOR 3202

LL6291-LOCAL LAW 62/91 - BOILERS 2748

LBLVIO-LOW PRESSURE BOILER 832

C-CONSTRUCTION 819

AEUHAZ1-FAIL TO CERTIFY CLASS 1 559

LL1081-LOCAL LAW 10/81 - ELEVATOR 374

ACC1-(OTHER BLDGS TYPES) - ELEVATOR AFFIRMATION OF CORRECTION 305

EVCAT1-ELEVATOR ANNUAL INSPECTION / TEST 235

BENCH-FAILURE TO BENCHMARK 120

LANDMK-LANDMARK 105

UB-UNSAFE BUILDINGS 68

P-PLUMBING 56

ES-ELECTRIC SIGNS 51

FISPNRF-NO REPORT AND / OR LATE FILING (FACADE) 47

EVCAT5-NON-RESIDENTIAL ELEVATOR PERIODIC INSPECTION/TEST 40

Z-ZONING 33

EARCX-FAILURE TO SUBMIT EER 32

JVIOS-PRIVATE RESIDENTIAL ELEVATOR 32

B-BOILER 31

L1198-LOCAL LAW 11/98 - FACADE 31

LL1198-LOCAL LAW 11/98 - FACADE 28

VCAT1-ELEVATOR ANNUAL INSPECTION / TEST 27

IMEGNCY-IMMEDIATE EMERGENCY 24

CMQ-MARQUEE 23

EGNCY-EMERGENCY 22

LL11/98-LOCAL LAW 11/98 - FACADE 20

LANDMRK-LANDMARK 18

FISP-FACADE SAFETY PROGRAM 16

LL1080-LOCAL LAW 10/80 - FACADE 14

LL10/80-LOCAL LAW 10/80 - FACADE 11

HBLVIO-HIGH PRESSURE BOILER 10

PA-PUBLIC ASSEMBLY 9

ACH1-(NYCHA) - ELEVATOR AFFIRMATION OF CORRECTION 9

RWNRF-RETAINING WALL 9

LL2604S-SPRINKLER 6

LL2604E-EMERGENCY POWER 5

JVCAT5-RESIDENTIAL ELEVATOR PERIODIC INSPECTION/TEST 5

LL5-LOCAL LAW 5/73 4

CS-SITE SAFETY 4

HVIOS-NYCHA ELEV ANNUAL INSPECTION/TEST 3

ACJ1-(PRIVATE RESIDENCE) - ELEVATOR AFFIRMATION OF CORRECTION 3

LL10/81-LOCAL LAW 10/81 - ELEVATOR 2

LL2604-PHOTOLUMINESCENT 2

LL16-LOCAL LAW 16/84 - ELEVATOR 2

CLOS-PADLOCK 2

A-SEU 1

HVCAT5-NYCHA ELEVATOR PERIODIC INSPECTION/TEST 1

Name: violation_type, dtype: int64

Most of the violations have to do with elevators and boilers.

I’m mainly interested in looking at the general types of violations, so here I will relabel any elevator-related violation as “ELEVATOR”, boiler violations as “BOILER” to make it cleaner.

That can be done with the where() function from NumPy. Docs.

The format is np.where(condition[, x, y]) .

It takes a boolean array, and then if an item in the array is True , yield x, else yield y.

elevators = df['violation_type'].str.contains('ELEVATOR')

boilers = df['violation_type'].str.contains('BOILER') df['violation_type'] = np.where(elevators,'ELEVATOR', np.where(boilers, 'BOILER', df['violation_type'])

Calling str.contains(substring) on a column returns a boolean mask of whether the row contains the substring .

You can also nest the np.where() calls, so, in this case, I nested the search for “BOILER” and then if neither “ELEVATOR” nor “BOILER” is found, just return the contents of df['violation_type'] .

Look at the value counts now.

In [8]: df['violation_type'].value_counts()

Out[8]:

ELEVATOR 4412

AEUHAZ1-FAIL TO CERTIFY CLASS 1 1856

BOILER 1550

BENCH-FAILURE TO BENCHMARK 993

C-CONSTRUCTION 394

FISPNRF-NO REPORT AND / OR LATE FILING (FACADE) 179

RWNRF-RETAINING WALL 138

LL2604S-SPRINKLER 107

EARCX-FAILURE TO SUBMIT EER 100

FISP-FACADE SAFETY PROGRAM 73

P-PLUMBING 72

IMEGNCY-IMMEDIATE EMERGENCY 40

LANDMK-LANDMARK 35

UB-UNSAFE BUILDINGS 19

HVIOS-NYCHA ELEV ANNUAL INSPECTION/TEST 14

EGNCY-EMERGENCY 10

Z-ZONING 4

CLOS-PADLOCK 3

CS-SITE SAFETY 1

Name: violation_type, dtype: int64

To further clean this up, I would want to research the other violation types like “AEUHAZ1-FAIL TO CERTIFY CLASS 1’”— it’s not clear what that is.

Sometimes you have to dig for more information about a dataset.