CSE 587: Deep Learning for Natural
Language Processing
Lecture 1. Introduction

Rui Zhang
Spring 2023

1

What is the goal of human language?

2

What is the goal of human language?
To share thoughts with conspecifics● to communicate with others● to store and pass knowledge3What is the goal of human language?
To share thoughts with conspecifics● to communicate with others● to store and pass knowledge

To think more complex thoughts● reasoning● interpretation

4

What is the goal of human language?
To share thoughts with conspecifics● to communicate with others● to store and pass knowledge

NoamChomsky

To think more complex thoughts● reasoning● interpretation

Almost all of your use of language isinternal. Virtually all of the use oflanguage has nothing to do withcommunication. The idea that language
has evolved as a system ofcommunication, or designed forcommunication, makes no sense.
…
The systems of thought use linguistic
expressions for reasoning, interpretation,organizing action, and other mental acts.
5What is the goal of human language?
To share thoughts with conspecifics● to communicate with others● to store and pass knowledge

NoamChomsky

Almost all of your use of language isinternal. Virtually all of the use oflanguage has nothing to do withcommunication. The idea that language
has evolved as a system ofcommunication, or designed forcommunication, makes no sense.
…
The systems of thought use linguistic
expressions for reasoning, interpretation,organizing action, and other mental acts.

To think more complex thoughts● reasoning● interpretation

The limits of mylanguage mean the limitsof my world.

LudwigWittgenstein
6Natural Language Processing (NLP)
It deals with natural languages that human speak (e.g., English, Chinese,Spanish, Arabic, etc), not programming languages.
● NLP is also referred to as Human Language Technology

7Natural Language Processing (NLP)
It deals with natural languages that human speak (e.g., English, Chinese,Spanish, Arabic, etc), not programming languages.
● NLP is also referred to as Human Language Technology
It aims to use computational approaches to analyze human languages.
● morphology, lexical analysis, syntax, semantics, discourse

8Natural Language Processing (NLP)
It deals with natural languages that human speak (e.g., English, Chinese,Spanish, Arabic, etc), not programming languages.
● NLP is also referred to as Human Language Technology
It aims to use computational approaches to analyze human languages.
● morphology, lexical analysis, syntax, semantics, discourse
It aims to build computer systems that can understand and generate naturallanguage.
● It can be used for human-human communication (e.g., machine translation,grammatical error correction) or human-machine communication (e.g., dialog
systems, question answering).
9Natural Language Processing vs Computational Linguistics
A lot of people use them interchangeably.
They are very similar, but also different.
The focus is a bit different:● Computational Linguistics focus on scientific questions in "Linguistics". It aims
to study scientific questions about human languages using computationaltools.
● Natural Language Processing focus on engineering problems of building
practical systems and applications that can mimic human usage of languages.

https://www.quora.com/How-is-computational-linguistics-different-from-natural-language-processing

10

Machine Translation

11

Question Answering

12Conversational Personal Assistants

Apple SiriGoogle Assistant

Microsoft Cortana13

Natural Language Understanding is Hard

14

Natural Language Understanding is Hard
"The old man the boat."15

Natural Language Understanding is Hard
"The old man the boat."

16

OpenAI GPT-3 Writes Entire Articles

17

GPT-3 as A Universal Language Model
https://beta.openai.com/?demo=118

Even for Writing Code...

19

NLP Tasks
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

Automatic speech recognition
CCG
Common sense
Constituency parsing
Coreference resolution
Data-to-Text Generation
Dependency parsing
Dialogue
Domain adaptation
Entity linking
Grammatical error correction
Information extraction
Intent Detection and Slot Filling
Language modeling
Lexical normalization
Machine translation

http://nlpprogress.com/

●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●

Missing elements
Multi-task learning
Multi-modal
Named entity recognition
Natural language inference
Part-of-speech tagging
Paraphrase Generation
Question answering
Relation prediction
Relationship extraction
Semantic textual similarity
Semantic parsing
Semantic role labeling
Sentiment analysis
Shallow syntax
Simplification
Stance detection
Summarization
Taxonomy learning
Temporal processing
Text classification
Word sense disambiguation

20

NLP Tasks
Foundational Technologies● Language Modeling● Part-of-speech Tagging● Syntactic Parsing● Dependency Parsing● Named Entity recognition● Coreference resolution● Word Sense Disambiguation● Semantic Role Labelling● …...

High-Level Tasks and Applications● Sentiment Analysis● Information Extraction● Machine Translation● Question Answering● Semantic Parsing● Summarization● Dialogue systems● Language and Vision● Data-to-Text Generation● …...
21

Natural Language Processing Systems
Input XTextTextText

Output YTaskLabel
Linguistic StructureTextText Classification (e.g., Sentiment Analysis)
Structured Prediction (e.g., Part-of-Speech Tagging)
Text Generation (e.g., Translation, Summarization)

22

Natural Language Processing Systems
Input X
Text
Text
Text

Output Y

Task

Label
Linguistic Structure
Text

Text Classification (e.g., Sentiment Analysis)
Structured Prediction (e.g., Part-of-Speech Tagging)
Text Generation (e.g., Translation, Summarization)

To create systems for X -> Y, we can● Expert Systems: Manually create rules mapping X to Y based on ourknowledge of languages● Data-Driven Methods: Collect Data and Use Machine Learning Techniques tolearn the mapping function
23

Data Driven Methods and Machine Learning“Machine Learning” is a data-driven problem solving strategy.

24

Data Driven Methods and Machine Learning“Machine Learning” is a data-driven problem solving strategy.
The ultimate goal of machine learning is generalization! Making future predictionon unseen examples as accurate as possible!
Training

Testing
Common theme is to solve a prediction problem:
Given an input,Predict an “appropriate” output (label, decision, action, etc).

25

Machine Learning Pipeline

26

Training, Validation, Test

27

Data Hygiene
Training Data: Use this and only this to train your neural networks.
Development Data: Use this to decide when to stop training your neuralnetworks, and do hyperparameter tuning.
Test Data: Use test data after picking your model. You should use test data onlyonce.

28

Neural Networks and Deep Learning

29

Deep Learning for NLP

From “Real-Time Social Media Analytics with Deep Transformer Language Models: A Big Data Approach”
by Ahmet and Abdullah

30

Course Topics
Part 1: Introduction and Neural Network Basics
1. Introduction to NLP
2. Text Classification and Language Modeling
3. Neural Networks and Backpropagation

Part 2: NLP Models
4. Sequence-to-Sequence, Attention, Transformers
5. Pretraining Language Models and Transfer Learning
6. Text-to-Text and Prompt-based Learning

Part 3: NLP Applications
7. Question Answering
8. Semantic Parsing
9. NLP for Programming
10. Natural Language Generation
11. Summarization
12. Multilingual NLP
13. Language and Vision
14. Commonsense Knowledge and Reasoning

Part 4: Advanced Techniques
15. Variational Autoencoders and Latent Variable Models
16. Structured Prediction
17. Robustness in NLP
18. Interpretability, Explainability, Model Analysis
19. Efficient NLP Methods
20. Ethics in NLP

31

Logistics

32

Course Information
LectureMoWe 2:30PM - 3:45PM @ Walker Building 124
Check our research!
Instructor
https://ryanzhumich.github.io/
Rui Zhang
Office Hour: Wednesdays 4-6pm @ W329 Westgate Building and Zoom
TAWenjian Miao
Office Hour: Thursdays 4-6pm @ W300
How to Contact us● For most questions, please use Piazza.
● Please only use Canvas Inbox for personal issues.
● Don't use psu emails: It's hard for me to monitor class status using emails.

33

Canvas Page
Canvas Inbox
PiazzaGradeScope

34

Textbooks
All Available online! You are strongly encouraged to read beyond slides.
●●
●
●
●
●
●Speech and Language Processing. Dan Jurafsky and James H. Martin.
Dive into Deep Learning. Aston Zhang, Zack C. Lipton, Mu Li, Alex J. Smola.
Natural Language Processing. Jacob Eisenstein.
Foundations of Statistical Natural Language Processing. Chris Manning andHinrich Schütze.
Linguistic Structure Prediction. Noah A. Smith.
Machine Learning: A Probabilistic Perspective. Kevin Murphy.
Linguistic Fundamentals for NLP. Emily M. Bender.

35

Prerequisites●CMPSC 448: Machine Learning●Programming in Python●CSE 582: Natural Language Processing

Previous experience in NLP/Deep Learning is preferred but not required.

36

Programming Assignment
One assignment on coding neural networks models for NLP.
● Minimalist Neural Networks for Sentiment Analysis● January 27 - February 13Python
GradeScope
Not late assignment accepted unless there is a compelling reason. Send me anCanvas Inbox if you know you are going to be late.

37

Paper Review and Presentation
Each student will do one lecture-style presentation (2 presentations in 1 class).
Each student will submit 5 paper reviews before the class and participate in thediscussion during presentations.

38

ExamsNo Exam. This course is a research project oriented course.

39

Project Format
Project Topics. This project aims to conduct original and independent researchover any NLP-related topics.
● Must deal with natural language data
● Must include some degree of model implementations and experiments
Group Policy. You can work on the course project in a group of 1 - 3 people. You
are allowed to combine this project with your research projects or projects fromother courses.

40

Project Deliverables●●●Project Proposal. Write a 3-page proposal that outlines your plan including
what problem or task you want to address, what dataset(s) you want to workon, what metrics you need to employ, what baselines you would like tocompare with. You should also cite a few relevant prior papers.
Final Report. Your final report should use our Latex template with at least
8-page plus references. Your report should begin with an abstract andintroduction to clearly state the problem you want to solve and contributions
you have made. It should also have a section on related work, a section on
your methodology, a section on your experimental settings and results, and asection on conclusions.
Code and Data. Please submit your data and runnable code with a detailedinstruction.
41

Project Timeline
Group Formation: January 27
Project Proposal: February 17
Final Report/Code and Data: May 5 (during the final exam week)42

Grading

Final letter grades will be curved and the cut scores will be determined after all grades
are finalized.

43