Enhancing Deep Reinforcement Learning
withCompressed Sensing-based State Estimation
Shaswot Shresthamali, Masaaki Kondo
Kondo Laboratory
Department of Information and Computer Science
Keio University

MCSoC 2023, Singapore
19-Dec-2023

Reinforcement Learning (RL)
• Sequential Decision-making• Feedback is provided using scalar rewards

2

Compressed Sensing + RL
• Adaptive control using RL has become very popular in the past years• IoT, AlphaGo/Star/Fold, Plasma Control, Scheduling jobs/power in embedded systems• RL suffer from missing state information due to• Sensor failures• Data transmission errors• Other noise sources• Can we recover performance of RL agents even in the presence of missing statedata?
• We use Compressed Sensing (CS) to recover/reconstruct missing sensor input
for robust RL.
3

The ViZDoom Environment• First person shooter game - BASIC scenario• Objective: shoot the monster• Actions• go-left,• go-right,• shoot• Rewards• -1 reward at each timestep• Successful hit: +101
• Miss: -5

Grayscale and Crop

• State space• Pixel information: grayscaled and cropped
4

Degradation due to missing pixels
Noisy Observation, noise = 0.2

Original Pixel Observation

Noisy Observation, noise = 0.8

Degradation
due to noise

noise = probability of pixel being
blacked out (i.e., pixel data ismissing)5

Image Compression: Review
DCT (DiscreteCosine Transform)

Keep only 20% of thelargest coefficients(Compress by 80%)
The original image can be reconstructed almost perfectly

Inverse DCT

6

Magic Behind Compressed Sensing

Image with missing data

Reconstructed Image

LinearAlgebra

Sparse DCT
coefficients infrequency domain

InverseDCTThe trick is to determine thesparse DCT coefficients fromthe undersampled image.

7

Compressed Sensing Problem Statement
𝚯 = 𝑪𝚿𝒚𝑪𝚿

• Infinitely large number of solutions• The trick• Find the sparsest solution• Use L1 regularization function• Better behavior than L0=

SubsampledInput

Random Compression
MatrixGiven

Known/Inferred𝒔

Underdetermined system oflinear equations

Let 𝚯 = 𝑪𝚿, thenDiscrete Cosine Transform
Known

Once we can obtain an estimate of 𝐬 as 𝐬ො , we can reconstruct anestimate of the original image by taking the inverse DCT.

where,Use LASSO (Least Absolute Shrinkage
and Selection Operator) to solve theoptimization problem and find 𝐬ො8Takeaway and Caveats𝐬ො𝒚Image with missing data

FlattenLASSO

𝚯 = 𝑪𝚿 must satisfy the Restricted Isometry Property (RIP)• 𝑪 and 𝚿 should be non-coherent• Usually satisfied if 𝑪 is a random sampling matrix• Distances between two signals are preserved after being
projected by 𝚯ෝ
𝒙Inverse
DCT

Reconstructed Image

Unflatten

If 𝐬 is K-sparse (i.e., it has K non-zero values), then thenumber of measurements 𝒑, should be sufficient large
on the order of9Proposed Framework: RL + CS
Observation with missing data

State Reconstruction using
Compressive Sensing

Actions
Reconstructed Observation

Actor-Critic RL Agent10

Results: Performance Recovery with CS• Agent trained in noise-free environment• Evaluated in noisy environment• Input recovery using lowpass filter and LASSO
Average reward innoise-free environment

Performance degradation
due to missing observation
data

Performance
Recovery using CS

11

Accelerating LASSO with Reduced Precision• By using high parallelization and reduced precision available in GPUs,
we can reduce the time for LASSO optimization significantly.

12

Results: RL + CS for Bipedal Walker• 4-joint walker robot environment• In this case, the state inputs are the hull angle speed, angular velocity, horizontal speed,
vertical speed, position of joints and joints angular speed, legs contact with ground, and 10
lidar rangefinder measurements.
• Noise is simulated by simply zeroing out some of the sensor values randomly.
• We maintain a history of previous observations for a certain time horizon [25 timesteps] forCS reconstruction.
13Conclusion• Performance of RL agents degrade significantly with observation noise
introduced due to missing data.
• We use CS to reconstruct the missing data and recover the performance of RL
agents in noisy environments. For sake of example, we use LASSO optimization.
• When input is relatively low dimensional, the computation for CS overhead isminimal.
• When input is high dimensional (e.g., pixels), CS requires large matrix
operations. Using GPUs and reduced precision can accelerate the optimization
process.
14

Shaswot SHRESTHAMALI

Masaaki KONDO

https://www.acsl.ics.keio.ac.jp/
www.shaswot.com

https://www.acsl.ics.keio.ac.jp/

15

Thank You
16