Automating Concurrent Negotiationsin Supply Chain Management
g-RIPS Sendai – NEC GroupTeam Members: Isaac Brown, Soto Hisakawa, Soichiro Sato, Kevin Zhou
Academic Mentor: Masaki Ogawa, Ph.D.
Industry Mentor: Yasser Mohammad, Ph.D.

Outline
1. Introduction
1.1.
1.2.

SCML Game
Problem/Goal

2. Reinforcement Learning
2.1. Background2.2. PPO
2.2.1. PPO agent
2.3. Q-Learning
2.3.1. Q-learning agent

3. Conclusion

OneShot SCM Framework●Agents each control a factory in the supply chain.

Image source: http://www.yasserm.com/scml/scml2020docs/tutorials/01.run_scml2020.html

OneShot SCM Framework●●

Agents each control a factory in the supply chain.
Each day in the SCM world, the following occurs:○ Exogenous contracts are distributed

Image source: http://www.yasserm.com/scml/scml2020docs/tutorials/01.run_scml2020.html

Level 0Level 1OneShot SCM Framework●●

Agents each control a factory in the supply chain.
Each day in the SCM world, the following occurs:○ Exogenous contracts are distributed○ Negotiations commence■ Negotiation issues:● Price● Quantity

Image source: http://www.yasserm.com/scml/scml2020docs/tutorials/01.run_scml2020.html

Level 0Level 1OneShot SCM Framework●●●

Agents each control a factory in the supply chain.
Each day in the SCM world, the following occurs:○ Exogenous contracts are distributed○ Negotiations commence■ Negotiation issues:● Price● Quantity
After X simulation days, agent with highest
percent proﬁt wins

Image source: http://www.yasserm.com/scml/scml2020docs/tutorials/01.run_scml2020.html

Level 0Level 1OneShot SCM Framework
●
●

●

●

Agents each control a factory in the supply chain.
Each day in the SCM world, the following occurs:○ Exogenous contracts are distributed○ Negotiations commence■ Negotiation issues:● Price● Quantity
After X simulation days, agent with highest
percent proﬁt wins

GOAL: Design an agent negotiation strategy
that performs well in the competition setting

Image source: http://www.yasserm.com/scml/scml2020docs/tutorials/01.run_scml2020.html

Level 0Level 1

Difﬁculties
GOAL: Design an agent negotiation strategy that performs well in the competition settingDifﬁculties
GOAL: Design an agent negotiation strategy that performs well in the competition setting
1.

Most world information is hidden to varying extents

Difﬁculties
GOAL: Design an agent negotiation strategy that performs well in the competition setting
1.

Most world information is hidden to varying extents
a. Opponent strategies
b. Opponent exogenous contractsc. All negotiation details not involving our agent

Difﬁculties
GOAL: Design an agent negotiation strategy that performs well in the competition setting
1.

2.

Most world information is hidden to varying extents
a. Opponent strategiesb. Opponent exogenous contractsc. All negotiation details not involving our agent
Many game characteristics are dynamic

Difﬁculties
GOAL: Design an agent negotiation strategy that performs well in the competition setting
1.

2.

Most world information is hidden to varying extents
a. Opponent strategies
b. Opponent exogenous contractsc. All negotiation details not involving our agent
Many game characteristics are dynamica. Trading prices
b. Exogenous contracts
c. Opponent strategies in some cases

Difﬁculties
GOAL: Design an agent negotiation strategy that performs well in the competition setting
1.

2.

3.

Most world information is hidden to varying extents
a. Opponent strategies
b. Opponent exogenous contractsc. All negotiation details not involving our agent
Many game characteristics are dynamica. Trading prices
b. Exogenous contracts
c. Opponent strategies in some cases
Attempts at developing agents without ML had mixed performanceDifﬁculties
GOAL: Design an agent negotiation strategy that performs well in the competition setting
1.

2.

3.

Most world information is hidden to varying extents
a. Opponent strategies
b. Opponent exogenous contractsc. All negotiation details not involving our agent
Many game characteristics are dynamica. Trading prices
b. Exogenous contracts
c. Opponent strategies in some cases
Attempts at developing agents without ML had mixed performancea. At worst, these agents were terrible
b. At best, these agents were good, but highly sensitive to the strategy environment they played inDifﬁculties
GOAL: Design an agent negotiation strategy that performs well in the competition setting
1.
2.
3.

Most world information is hidden to varying extents
Many game characteristics are dynamic
Attempts at developing agents without ML had mixed performance

FIX: Use reinforcement learning to learn how to play against a variety of opposing negotiation strategies
and world environments.

Reinforcement Learning (RL)

Typical Reinforcement Learning cycle
Image source: https://towardsdatascience.com/reinforcement-learning-101-e24b50e1d292

Reinforcement Learning (RL)
Some key terms that describe the basic elements of an RL are:
●
●
●
●
●
●

Agent ー A decision-making and action subject.
Environment ー Physical world in which the agent operates
State ー Current situation of the agent
Action ー The behavior of the agent
Reward ー Feedback from the environment
Policy ー Method to map agent’s state to actions

Why use RL?
Features :●●Information (states, actions, rewards) is sent and received between the agent
and the environment.
It uses its own observations (states and rewards) to update its policies, learn
better behaviors, and maximize long-term beneﬁts.

Beneﬁts :●Agents can obtain information about their environment as they act.

●Leads to the discovery of new strategies.

●

Effective for parameter optimization.

RL Agent Development
Issues :●●It has drawbacks such as increased training time due to excessive state
information and the possibility of overlearning.
The performance may deteriorate due to insufﬁcient state information.

It’s necessary to appropriately select a large amount of state information (e.g.,rounds, unit price, quantity, etc.) for agent development in RL.

Algorithms for RL :●Use Proximal Policy Optimization(PPO) and Q-learning

Proximal Policy Optimization Algorithm

Proximal Policy Optimization Algorithm●An advantage function A𝜋: S×A → ℝ is a mapping from state-action pairs to real values according toA𝜋(s,a) = Q𝜋(s,a) − V𝜋(s)where Q𝜋(s,a) is a measure of the quality of a state-action pair and V𝜋(s) is a measure of the averagevalue obtained at state s, all with respect to policy 𝜋.

Proximal Policy Optimization Algorithm●An advantage function A𝜋: S×A → ℝ is a mapping from state-action pairs to real values according toA𝜋(s,a) = Q𝜋(s,a) − V𝜋(s)where Q𝜋(s,a) is a measure of the quality of a state-action pair and V𝜋(s) is a measure of the averagevalue obtained at state s, all with respect to policy 𝜋.

●Algorithm:○ N agents are given the same policy 𝜋𝜃 to use in parallel environments for T steps

Proximal Policy Optimization Algorithm●An advantage function A𝜋: S×A → ℝ is a mapping from state-action pairs to real values according toA𝜋(s,a) = Q𝜋(s,a) − V𝜋(s)where Q𝜋(s,a) is a measure of the quality of a state-action pair and V𝜋(s) is a measure of the averagevalue obtained at state s, all with respect to policy 𝜋.

●Algorithm:○ N agents are given the same policy 𝜋𝜃 to use in parallel environments for T steps○ At each timestep t ∈ [1, T] and for each agent, approximate an advantage function ÂtProximal Policy Optimization Algorithm●An advantage function A𝜋: S×A → ℝ is a mapping from state-action pairs to real values according toA𝜋(s,a) = Q𝜋(s,a) − V𝜋(s)where Q𝜋(s,a) is a measure of the quality of a state-action pair and V𝜋(s) is a measure of the averagevalue obtained at state s, all with respect to policy 𝜋.

●Algorithm:○ N agents are given the same policy 𝜋𝜃 to use in parallel environments for T steps○ At each timestep t ∈ [1, T] and for each agent, approximate an advantage function Ât○ Optimize loss function with respect to policy parameters 𝜃 using the advantage function set dataProximal Policy Optimization Algorithm●Algorithm:○ N agents are given the same policy 𝜋𝜃 to use in parallel environments for T steps○ At each timestep t ∈ [1, T] and for each agent, approximate an advantage function Ât○ Optimize loss function with respect to policy parameters 𝜃 using the advantage function set data
○ Main difference between PPO and other policy gradient methods is the loss function:
LCLIP = 𝔼t[clip(rt(𝜃), 1−𝜀, 1+𝜀) ∙ Ât]

where rt(𝜃) measures the “size” of the update to thepolicy 𝜋𝜃.

Image Source: https://arxiv.org/pdf/1707.06347.pdf

SCML and RL
The most recent version of SCML implements an interface for using reinforcement
learning to train agentsSome aspects of the interface can (and should) be customized to improve performanceUtilizing PPO - Deciding the Observation Space
During the course of an SCML world, a lot of information is available to agents

Need to decide what is useful for the agent to observe:●●

Tradeoff: performance vs training time
Possibility for overﬁtting to low-relevance information

Can also incorporate negotiation history into observations

Utilizing PPO - Reward Shaping
Another approach: customize the reward given to the agent to encourage “good”
behavior

Default reward is based on proﬁt●●Proﬁt is only received at the end of a day
Proﬁt is dependent on random world variables

One possible solution: reward the agent for reducing its needs

Utilizing PPO - Training Environment
Default training environment places the agent in a world populated by “default”
agents that are present in tournament simulations

Can also train in environments that include other custom agents – this simulates
tournament conditions and allows training against agents with known goodperformance

PPO Agents Results – reward shaping

An agent whose reward primarily depends on proﬁtPPO Agents Results – reward shaping

An agent whose reward only depends on quantity

Additional work – training & testing scripts
Initial results indicate that performance is still poor – much more experimentationand iteration is needed

To facilitate this, we have developed scripts for easy testing and training

Q-learning method (1/3) - reduce information
In this game, information space is massive●The number of choices when making proposals:(2 × 10)number of partners → 2 × 10 or lessSolution: Use same proposal for all partners (avoiding bad actions very important!)●Opponent offer information received which we use for reinforcement learning(2 × 10)number of partners → 0
Solution: Use a ﬁxed acceptance strategy

Q-learning method (2/3) - how ﬁxed acceptance strategy?
How ﬁxed acceptance strategy?
●

In level 0 or step ≥ 19
→ the best price combination of offers which the most fulﬁll needs●

In level 1 and step < 19
→ the most offers fulﬁlling needs which have the best priceAnd, if remaining needs and partners, propose counter offer. Else, end negotiation.

Q-learning method (3/3) - treat state information So, we should estimate only propose unit price and propose quantity

Selected state information for learning= current step, current needs, level, current number of negotiating partners

Relative to propose unit price

Relative to propose quantity

Number of combination of selected state information for learning = 3360

My method of Q-learning (4/4)
Learning environment
n_steps = 50
Competitor = [Me, 1st place, 2nd place, 3rd place]
Why Q-learning●It is easy to make and adjust

All action is random, learning rate = 20, discount factor = 999/1000

Analysis (1/3) - new agent by learning
mean

medianCCAgent

1.086311.07852

QuantityOrientedAgent

1.08546

1.08176

AgentByQValue

1.08299

1.07234

KanbeAgent

1.080641.07855

The reason for losing seems the action is
not consistent for lack of learning
But, we get strategy that seems best through analysis tendency of best action bylearning. So, tried to make new Agent from result of the analysis

Analysis (2/3) - best propose quantity
Average of best propose quantity by learning

needs

123
4
5
678910112334556*
6*5*Number of partners
23451111*1211*22223232333344*
4344*
4*
44*
5*
546*
6*
455*
7*
76*: seems to lack of learning Signiﬁcantly

61*
1*
2*
2*
333466If number of partners = 1
Propose quantity = Needs
Elif needs ≤ 9
Propose quantity =Needs/2

Else
Propose quantity = Needs/2 + 1

Analysis (3/3) - best propose unit price

Most of the time,
best unit price is better
than worst unit price inboth level

Propose unit price = best unit priceResult (1/2) - compare with 2023 winners
mean

medianTestAgent0730

1.10879

1.10639

KanbeAgent

1.10058

1.08732

CCAgent

1.09829

1.09755

QuantityOrientedAgent

1.09726

1.09598

Our agent won against agents to betrained on mean and medium (but, ouragent lost on minimum)

Against other agents which are not to be trained?

Result (2/2) - compare with 2023 other Finalist
mean

medianAgentVSCforOneShot 1.11868

1.1192

TestAgent0730

1.086211.08625

Shochan

1.077641.08939

ForestAgent

1.06246

1.07911PHLA

1.03695

1.0493AgentSAS

1.03627

1.05069

NegoAgent

1.02836

1.03822My Agent lost to speciﬁc agent(AgentVSCforOneShot) alltime

Considerations (1/2)- why lost?

Difference between fulﬁlling needs and not fulﬁlling needs especially in level 1

Considerations (2/2)
Make new agent TestAgent0803 which have two difference from TestAgent0730 inlevel 1●●change the number of steps to completely compromise the unit price from 19 to18
compromise the unit price up to permit_needs per day even if the number ofsteps is less than 18

At ﬁrst, permit needs is 0. And, we renew permit_needs as below every 10 days
based on the average of remaining ﬁnal needs for the last 10 day
permit_needs → min{permin_needs+1, 5}(the average > 0.7)

permit_needs → max{permin_needs -1, 0}(the average < 0.3)

New result (1/2) - compare with AgentSCVforOneShot

mean

medianAgentSCVforOneShot

1.1097

1.1149

TestAgent0730

1.1005

1.0967

mean

medianTestAgent0803

1.11291.1137

AgentSCVforOneShot

1.1127

1.1136

New result (2/2) - compare with 2023 winners
meanmedian

TestAgent0803

1,12201.1038CCAgent

1.11691.1098KanbeAgent

1.1135

1.1042

QuantityOrientedAgent

1.10801.093

Of course, TestAgent0803
can take good performanceagainst 2023 winners.

Future work (Q-learning)●●●

Consider level-dependant acceptance strategies that are more adaptable to theenvironment
Optimize each parameter
Use better hardware in order to evaluate and optimize each parameter

ConclusionDeveloping agents for SCML is very difﬁcult!

Open-ended RL achieves modest performance – more tuning and iteration may allow
it to achieve much better results

Mixed RL and hardcoded strategies achieve good results without needing a great
deal of engineering