Repository: alteryx/woodwork
Pull request #1210 (2021-12-02 07:33:05)

Add Infer Natural Language to Woodwork
- Fixes https://github.com/alteryx/woodwork/issues/1211

codecov[bot]: # [Codecov](https://codecov.io/gh/alteryx/woodwork/pull/1210?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) Report
> Merging [#1210](https://codecov.io/gh/alteryx/woodwork/pull/1210?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (2f4fb8e) into [main](https://codecov.io/gh/alteryx/woodwork/commit/b38dcfbadf467a7dc8b476192065f331a094e9d4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (b38dcfb) will **decrease** coverage by `3.81%`.
> The diff coverage is `97.14%`.

[![Impacted file tree graph](https://codecov.io/gh/alteryx/woodwork/pull/1210/graphs/tree.svg?width=650&height=150&src=pr&token=KJCKMREBDP&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx)](https://codecov.io/gh/alteryx/woodwork/pull/1210?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx)

```diff
@@            Coverage Diff             @@
##             main    #1210      +/-   ##
==========================================
- Coverage   99.95%   96.13%   -3.82%     
==========================================
  Files          49       49              
  Lines        8530     8564      +34     
==========================================
- Hits         8526     8233     -293     
- Misses          4      331     +327     
```


| [Impacted Files](https://codecov.io/gh/alteryx/woodwork/pull/1210?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) | Coverage Δ | |
|---|---|---|
| [woodwork/type\_sys/type\_system.py](https://codecov.io/gh/alteryx/woodwork/pull/1210/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdHlwZV9zeXMvdHlwZV9zeXN0ZW0ucHk=) | `95.71% <ø> (-4.29%)` | :arrow_down: |
| [woodwork/tests/type\_system/test\_ltype\_inference.py](https://codecov.io/gh/alteryx/woodwork/pull/1210/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdHlwZV9zeXN0ZW0vdGVzdF9sdHlwZV9pbmZlcmVuY2UucHk=) | `90.64% <87.50%> (-9.36%)` | :arrow_down: |
| [woodwork/tests/type\_system/conftest.py](https://codecov.io/gh/alteryx/woodwork/pull/1210/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdHlwZV9zeXN0ZW0vY29uZnRlc3QucHk=) | `98.27% <100.00%> (-1.73%)` | :arrow_down: |
| [woodwork/type\_sys/inference\_functions.py](https://codecov.io/gh/alteryx/woodwork/pull/1210/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdHlwZV9zeXMvaW5mZXJlbmNlX2Z1bmN0aW9ucy5weQ==) | `100.00% <100.00%> (ø)` | |
| [woodwork/tests/conftest.py](https://codecov.io/gh/alteryx/woodwork/pull/1210/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvY29uZnRlc3QucHk=) | `82.15% <0.00%> (-17.85%)` | :arrow_down: |
| [woodwork/typing.py](https://codecov.io/gh/alteryx/woodwork/pull/1210/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdHlwaW5nLnB5) | `83.33% <0.00%> (-16.67%)` | :arrow_down: |
| [woodwork/tests/testing\_utils/table\_utils.py](https://codecov.io/gh/alteryx/woodwork/pull/1210/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdGVzdGluZ191dGlscy90YWJsZV91dGlscy5weQ==) | `83.33% <0.00%> (-16.67%)` | :arrow_down: |
| [woodwork/serialize.py](https://codecov.io/gh/alteryx/woodwork/pull/1210/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvc2VyaWFsaXplLnB5) | `88.28% <0.00%> (-11.72%)` | :arrow_down: |
| [woodwork/tests/logical\_types/test\_logical\_types.py](https://codecov.io/gh/alteryx/woodwork/pull/1210/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvbG9naWNhbF90eXBlcy90ZXN0X2xvZ2ljYWxfdHlwZXMucHk=) | `88.69% <0.00%> (-11.31%)` | :arrow_down: |
| [woodwork/tests/accessor/test\_indexers.py](https://codecov.io/gh/alteryx/woodwork/pull/1210/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvYWNjZXNzb3IvdGVzdF9pbmRleGVycy5weQ==) | `90.37% <0.00%> (-9.63%)` | :arrow_down: |
| ... and [19 more](https://codecov.io/gh/alteryx/woodwork/pull/1210/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/alteryx/woodwork/pull/1210?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/alteryx/woodwork/pull/1210?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx). Last update [b38dcfb...2f4fb8e](https://codecov.io/gh/alteryx/woodwork/pull/1210?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx).


Patch:
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index 661fe55..013beec 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -2,15 +2,17 @@
 
 Release Notes
 -------------
-.. Future Release
-  ==============
+Future Release
+==============
     * Enhancements
+        * Add type inference for natural language (:pr:`1210`) 
     * Fixes
     * Changes
     * Documentation Changes
     * Testing Changes
 
-.. Thanks to the following people for contributing to this release:
+   Thanks to the following people for contributing to this release:
+   :user:`tuethan1999`
 
 v0.10.0 Nov 30, 2020
 ====================
diff --git a/setup.py b/setup.py
index d4ce6d7..3a92f21 100644
--- a/setup.py
+++ b/setup.py
@@ -40,5 +40,6 @@ setup(
         ]
     },
     long_description=long_description,
-    long_description_content_type='text/markdown'
+    long_description_content_type='text/markdown',
+    data_files=[('woodwork/data', ['woodwork/data/1-1000.txt'])]
 )
diff --git a/woodwork/data/1-1000.txt b/woodwork/data/1-1000.txt
new file mode 100644
index 0000000..f10b1af
--- /dev/null
+++ b/woodwork/data/1-1000.txt
@@ -0,0 +1,1000 @@
+the
+of
+to
+and
+a
+in
+is
+it
+you
+that
+he
+was
+for
+on
+are
+with
+as
+I
+his
+they
+be
+at
+one
+have
+this
+from
+or
+had
+by
+not
+word
+but
+what
+some
+we
+can
+out
+other
+were
+all
+there
+when
+up
+use
+your
+how
+said
+an
+each
+she
+which
+do
+their
+time
+if
+will
+way
+about
+many
+then
+them
+write
+would
+like
+so
+these
+her
+long
+make
+thing
+see
+him
+two
+has
+look
+more
+day
+could
+go
+come
+did
+number
+sound
+no
+most
+people
+my
+over
+know
+water
+than
+call
+first
+who
+may
+down
+side
+been
+now
+find
+any
+new
+work
+part
+take
+get
+place
+made
+live
+where
+after
+back
+little
+only
+round
+man
+year
+came
+show
+every
+good
+me
+give
+our
+under
+name
+very
+through
+just
+form
+sentence
+great
+think
+say
+help
+low
+line
+differ
+turn
+cause
+much
+mean
+before
+move
+right
+boy
+old
+too
+same
+tell
+does
+set
+three
+want
+air
+well
+also
+play
+small
+end
+put
+home
+read
+hand
+port
+large
+spell
+add
+even
+land
+here
+must
+big
+high
+such
+follow
+act
+why
+ask
+men
+change
+went
+light
+kind
+off
+need
+house
+picture
+try
+us
+again
+animal
+point
+mother
+world
+near
+build
+self
+earth
+father
+head
+stand
+own
+page
+should
+country
+found
+answer
+school
+grow
+study
+still
+learn
+plant
+cover
+food
+sun
+four
+between
+state
+keep
+eye
+never
+last
+let
+thought
+city
+tree
+cross
+farm
+hard
+start
+might
+story
+saw
+far
+sea
+draw
+left
+late
+run
+don't
+while
+press
+close
+night
+real
+life
+few
+north
+open
+seem
+together
+next
+white
+children
+begin
+got
+walk
+example
+ease
+paper
+group
+always
+music
+those
+both
+mark
+often
+letter
+until
+mile
+river
+car
+feet
+care
+second
+book
+carry
+took
+science
+eat
+room
+friend
+began
+idea
+fish
+mountain
+stop
+once
+base
+hear
+horse
+cut
+sure
+watch
+color
+face
+wood
+main
+enough
+plain
+girl
+usual
+young
+ready
+above
+ever
+red
+list
+though
+feel
+talk
+bird
+soon
+body
+dog
+family
+direct
+pose
+leave
+song
+measure
+door
+product
+black
+short
+numeral
+class
+wind
+question
+happen
+complete
+ship
+area
+half
+rock
+order
+fire
+south
+problem
+piece
+told
+knew
+pass
+since
+top
+whole
+king
+space
+heard
+best
+hour
+better
+true
+during
+hundred
+five
+remember
+step
+early
+hold
+west
+ground
+interest
+reach
+fast
+verb
+sing
+listen
+six
+table
+travel
+less
+morning
+ten
+simple
+several
+vowel
+toward
+war
+lay
+against
+pattern
+slow
+center
+love
+person
+money
+serve
+appear
+road
+map
+rain
+rule
+govern
+pull
+cold
+notice
+voice
+unit
+power
+town
+fine
+certain
+fly
+fall
+lead
+cry
+dark
+machine
+note
+wait
+plan
+figure
+star
+box
+noun
+field
+rest
+correct
+able
+pound
+done
+beauty
+drive
+stood
+contain
+front
+teach
+week
+final
+gave
+green
+oh
+quick
+develop
+ocean
+warm
+free
+minute
+strong
+special
+mind
+behind
+clear
+tail
+produce
+fact
+street
+inch
+multiply
+nothing
+course
+stay
+wheel
+full
+force
+blue
+object
+decide
+surface
+deep
+moon
+island
+foot
+system
+busy
+test
+record
+boat
+common
+gold
+possible
+plane
+stead
+dry
+wonder
+laugh
+thousand
+ago
+ran
+check
+game
+shape
+equate
+hot
+miss
+brought
+heat
+snow
+tire
+bring
+yes
+distant
+fill
+east
+paint
+language
+among
+grand
+ball
+yet
+wave
+drop
+heart
+am
+present
+heavy
+dance
+engine
+position
+arm
+wide
+sail
+material
+size
+vary
+settle
+speak
+weight
+general
+ice
+matter
+circle
+pair
+include
+divide
+syllable
+felt
+perhaps
+pick
+sudden
+count
+square
+reason
+length
+represent
+art
+subject
+region
+energy
+hunt
+probable
+bed
+brother
+egg
+ride
+cell
+believe
+fraction
+forest
+sit
+race
+window
+store
+summer
+train
+sleep
+prove
+lone
+leg
+exercise
+wall
+catch
+mount
+wish
+sky
+board
+joy
+winter
+sat
+written
+wild
+instrument
+kept
+glass
+grass
+cow
+job
+edge
+sign
+visit
+past
+soft
+fun
+bright
+gas
+weather
+month
+million
+bear
+finish
+happy
+hope
+flower
+clothe
+strange
+gone
+jump
+baby
+eight
+village
+meet
+root
+buy
+raise
+solve
+metal
+whether
+push
+seven
+paragraph
+third
+shall
+held
+hair
+describe
+cook
+floor
+either
+result
+burn
+hill
+safe
+cat
+century
+consider
+type
+law
+bit
+coast
+copy
+phrase
+silent
+tall
+sand
+soil
+roll
+temperature
+finger
+industry
+value
+fight
+lie
+beat
+excite
+natural
+view
+sense
+ear
+else
+quite
+broke
+case
+middle
+kill
+son
+lake
+moment
+scale
+loud
+spring
+observe
+child
+straight
+consonant
+nation
+dictionary
+milk
+speed
+method
+organ
+pay
+age
+section
+dress
+cloud
+surprise
+quiet
+stone
+tiny
+climb
+cool
+design
+poor
+lot
+experiment
+bottom
+key
+iron
+single
+stick
+flat
+twenty
+skin
+smile
+crease
+hole
+trade
+melody
+trip
+office
+receive
+row
+mouth
+exact
+symbol
+die
+least
+trouble
+shout
+except
+wrote
+seed
+tone
+join
+suggest
+clean
+break
+lady
+yard
+rise
+bad
+blow
+oil
+blood
+touch
+grew
+cent
+mix
+team
+wire
+cost
+lost
+brown
+wear
+garden
+equal
+sent
+choose
+fell
+fit
+flow
+fair
+bank
+collect
+save
+control
+decimal
+gentle
+woman
+captain
+practice
+separate
+difficult
+doctor
+please
+protect
+noon
+whose
+locate
+ring
+character
+insect
+caught
+period
+indicate
+radio
+spoke
+atom
+human
+history
+effect
+electric
+expect
+crop
+modern
+element
+hit
+student
+corner
+party
+supply
+bone
+rail
+imagine
+provide
+agree
+thus
+capital
+won't
+chair
+danger
+fruit
+rich
+thick
+soldier
+process
+operate
+guess
+necessary
+sharp
+wing
+create
+neighbor
+wash
+bat
+rather
+crowd
+corn
+compare
+poem
+string
+bell
+depend
+meat
+rub
+tube
+famous
+dollar
+stream
+fear
+sight
+thin
+triangle
+planet
+hurry
+chief
+colony
+clock
+mine
+tie
+enter
+major
+fresh
+search
+send
+yellow
+gun
+allow
+print
+dead
+spot
+desert
+suit
+current
+lift
+rose
+continue
+block
+chart
+hat
+sell
+success
+company
+subtract
+event
+particular
+deal
+swim
+term
+opposite
+wife
+shoe
+shoulder
+spread
+arrange
+camp
+invent
+cotton
+born
+determine
+quart
+nine
+truck
+noise
+level
+chance
+gather
+shop
+stretch
+throw
+shine
+property
+column
+molecule
+select
+wrong
+gray
+repeat
+require
+broad
+prepare
+salt
+nose
+plural
+anger
+claim
+continent
+oxygen
+sugar
+death
+pretty
+skill
+women
+season
+solution
+magnet
+silver
+thank
+branch
+match
+suffix
+especially
+fig
+afraid
+huge
+sister
+steel
+discuss
+forward
+similar
+guide
+experience
+score
+apple
+bought
+led
+pitch
+coat
+mass
+card
+band
+rope
+slip
+win
+dream
+evening
+condition
+feed
+tool
+total
+basic
+smell
+valley
+nor
+double
+seat
+arrive
+master
+track
+parent
+shore
+division
+sheet
+substance
+favor
+connect
+post
+spend
+chord
+fat
+glad
+original
+share
+station
+dad
+bread
+charge
+proper
+bar
+offer
+segment
+slave
+duck
+instant
+market
+degree
+populate
+chick
+dear
+enemy
+reply
+drink
+occur
+support
+speech
+nature
+range
+steam
+motion
+path
+liquid
+log
+meant
+quotient
+teeth
+shell
+neck
\ No newline at end of file
diff --git a/woodwork/type_sys/inference_functions.py b/woodwork/type_sys/inference_functions.py
index bbf7726..1e23949 100644
--- a/woodwork/type_sys/inference_functions.py
+++ b/woodwork/type_sys/inference_functions.py
@@ -1,11 +1,22 @@
-from typing import Callable
+import importlib.resources as pkg_resources
+from typing import Any, Callable, Iterable, Union
 
+import numpy as np
 import pandas as pd
 import pandas.api.types as pdtypes
 
 import woodwork as ww
+from woodwork import data
 from woodwork.type_sys.utils import _is_categorical_series, col_is_datetime
 
+Tokens = Iterable[str]
+
+COMMON_WORDS_SET = set(
+    line.strip().lower() for line in pkg_resources.open_text(data, "1-1000.txt")
+)
+
+NL_delimiters = r"[- \[\].,!\?;\n]"
+
 
 def categorical_func(series):
     if pdtypes.is_categorical_dtype(series.dtype):
@@ -82,6 +93,25 @@ def timedelta_func(series):
     return False
 
 
+def num_common_words(wordlist: Union[Tokens, Any]) -> float:
+    if not isinstance(wordlist, Iterable):
+        return np.nan
+    num_common_words = 0
+    for x in wordlist:
+        if x.lower() in COMMON_WORDS_SET:
+            num_common_words += 1
+    return num_common_words
+
+
+def natural_language_func(series):
+    tokens = series.astype("string").str.split(NL_delimiters)
+    mean_num_common_words = np.nanmean(tokens.map(num_common_words))
+
+    return (
+        mean_num_common_words > 1.14
+    )  # determined through https://github.com/alteryx/nl_inference
+
+
 class InferWithRegex:
     def __init__(self, get_regex: Callable[[], str]):
         self.get_regex = get_regex
diff --git a/woodwork/type_sys/type_system.py b/woodwork/type_sys/type_system.py
index aebda5d..7ae9eec 100644
--- a/woodwork/type_sys/type_system.py
+++ b/woodwork/type_sys/type_system.py
@@ -10,6 +10,7 @@ from .inference_functions import (
     integer_func,
     integer_nullable_func,
     ip_address_func,
+    natural_language_func,
     timedelta_func,
     url_func,
 )
@@ -62,7 +63,7 @@ DEFAULT_INFERENCE_FUNCTIONS = {
     IntegerNullable: integer_nullable_func,
     IPAddress: ip_address_func,
     LatLong: None,
-    NaturalLanguage: None,
+    NaturalLanguage: natural_language_func,
     Ordinal: None,
     PhoneNumber: None,
     PostalCode: None,


Test patch:
diff --git a/woodwork/data/__init__.py b/woodwork/data/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/woodwork/tests/type_system/conftest.py b/woodwork/tests/type_system/conftest.py
index 93aa3be..a970e8c 100644
--- a/woodwork/tests/type_system/conftest.py
+++ b/woodwork/tests/type_system/conftest.py
@@ -235,6 +235,41 @@ def timedeltas(request):
     return request.getfixturevalue(request.param)
 
 
+# Natural Language Fixtures
+@pytest.fixture
+def pandas_natural_language():
+    return [
+        pd.Series(
+            [
+                "Hello World! My name is bob!",
+                "I like to move it move it",
+                "its cold outside",
+            ]
+        ),
+    ]
+
+
+@pytest.fixture
+def dask_natural_language(pandas_natural_language):
+    return [pd_to_dask(series) for series in pandas_natural_language]
+
+
+@pytest.fixture
+def koalas_natural_language(pandas_natural_language):
+    return [pd_to_koalas(series) for series in pandas_natural_language]
+
+
+@pytest.fixture(
+    params=[
+        "pandas_natural_language",
+        "dask_natural_language",
+        "koalas_natural_language",
+    ]
+)
+def natural_language(request):
+    return request.getfixturevalue(request.param)
+
+
 # Unknown Inference Fixtures
 @pytest.fixture
 def pandas_strings():
@@ -264,6 +299,14 @@ def strings(request):
 @pytest.fixture
 def pandas_pdnas():
     return [
+        pd.Series(
+            [
+                "Hello World! My name is bob!",
+                pd.NA,
+                "I like to move it move it",
+                "its cold outside",
+            ]
+        ),
         pd.Series(["Mr. John Doe", pd.NA, "James Brown", "Ms. Paige Turner"]).astype(
             "string"
         ),
diff --git a/woodwork/tests/type_system/test_ltype_inference.py b/woodwork/tests/type_system/test_ltype_inference.py
index 6fae7f2..a12ee37 100644
--- a/woodwork/tests/type_system/test_ltype_inference.py
+++ b/woodwork/tests/type_system/test_ltype_inference.py
@@ -10,6 +10,7 @@ from woodwork.logical_types import (
     Integer,
     IntegerNullable,
     LogicalType,
+    NaturalLanguage,
     Timedelta,
     Unknown,
 )
@@ -114,6 +115,16 @@ def test_categorical_inference(categories):
             assert isinstance(inferred_type, Categorical)
 
 
+def test_natural_language_inference(natural_language):
+    dtypes = ["object", "string"]
+    if _is_koalas_series(natural_language[0]):
+        dtypes = get_koalas_dtypes(dtypes)
+    for series in natural_language:
+        for dtype in dtypes:
+            inferred_type = ww.type_system.infer_logical_type(series.astype(dtype))
+            assert isinstance(inferred_type, NaturalLanguage)
+
+
 def test_categorical_inference_based_on_dtype(categories_dtype):
     """
     This test specifically targets the case in which a series can be inferred
@@ -182,6 +193,7 @@ def test_unknown_inference_all_null(nulls):
 
 def test_pdna_inference(pdnas):
     expected_logical_types = [
+        NaturalLanguage,
         Unknown,
         IntegerNullable,
         BooleanNullable,