Q:

Python - textual analysis - search for phrases instead of simple words (tokens)

I have a document in python which I read in the following way:
tokens = re.findall('\w+', doc)  # Note that \w+ splits hyphenated words
for token in tokens:

The problem with this is that I can search for simple words (e.g. "bananas", but not for phrases, e.g. "yellow bananas").  How should I change my code in order to be able to search for phrases on top of simple words? 
Seems like I somehow need to read the whole file as a string instead of tokenizing it word by word? 

A:

You seem to want to get the words and then join them with space, since your "phrases"  seem to be single space separated words.  Use 
text = ''
with open(path, 'r') as fr:
    text = fr.read()
clean_text = " ".join(re.findall(r'\w+(?:-\w+)*', text)) 

Then, you may simply use .find(). 
The \w+(?:-\w+)* pattern extracts both hyphenated and non-hyphenated words:

\w+ - one or more letters, digits, _s
(?: -\w+)* - 0 or more repetitions of

- - a hyphen

\w+ - one or more letters, digits, _s.

