prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What was the club known as before being officially renamed FC Urartu?

Read the following text and answer briefly.

Context 1: Football Club Urartu (, translated Futbolayin Akumb Urartu), commonly known as Urartu, is an Armenian professional football team based in the capital Yerevan that currently plays in the Armenian Premier League  The club won the Armenian Cup three times, in 1992, 2007 and 2016  In 2013–2014, they won the Armenian Premier League for the first time in their history 

In early 2016, the Russia-based Armenian businessman Dzhevan Cheloyants became a co-owner of the club after purchasing the major part of the club shares  The club was known as FC Banants until 1 August 2019, when it was officially renamed FC Urartu 

History

Kotayk
Urartu FC were founded as FC Banants by Sarkis Israelyan on 21 January 1992 in the village of Kotayk, representing the Kotayk Province  He named the club after his native village of Banants (currently known as Bayan)  Between 1992 and 1995, the club was commonly referred to as Banants Kotayk  During the 1992 season, the club won the first Armenian Cup  At the end of the 1995 transitional season, Banants suffered a financial crisis  The club owners decided that it was better to merge the club with FC Kotayk of Abovyan, rather than disband it  In 2001, Banants demerged from FC Kotayk, and was moved from Abovyan to the capital Yerevan 

Yerevan

FC Banants was relocated to Yerevan in 2001  At the beginning of 2003, Banants merged with FC Spartak Yerevan, but was able to limit the name of the new merger to FC Banants  Spartak became Banants's youth academy and later changed the name to Banants-2  Because of the merger, Banants acquired many players from Spartak Yerevan, including Samvel Melkonyan  After the merger, Banants took a more serious approach and have finished highly in the league table ever since  The club managed to lift the Armenian Cup in 2007 
Experience is making way for youth for the 2008 and 2009 seasons  The departures of most of the experienced players have left the club's future to the youth  Along with two Ukrainian players, Ugandan international, Noah Kasule, has been signed 

The club headquarters are located on Jivani Street 2 of the Malatia-Sebastia District, Yerevan 

Domestic

European

Stadium

The construction of the Banants Stadium was launched in 2006 in the Malatia-Sebastia District of Yerevan, with the assistance of the FIFA goal programme  It was officially opened in 2008 with a capacity of 3,600 seats  Further developments were implemented later in 2011, when the playing pitch was modernized and the capacity of the stadium was increased up to 4,860 seats (2,760 at the northern stand, 1,500 at the southern stand and 600 at the western stand) 

Training centre/academy
Banants Training Centre is the club's academy base located in the Malatia-Sebastia District of Yerevan  In addition to the main stadium, the centre houses 3 full-size training pitches, mini football pitches as well as an indoor facility  The current technical director of the academy is the former Russian footballer Ilshat Faizulin 

Fans
The most active group of fans is the South West Ultras fan club, mainly composed of residents from several neighbourhoods within the Malatia-Sebastia District of Yerevan, since the club is a de facto representer of the district  Members of the fan club benefit from events organized by the club and many facilities of the Banants training centre, such as the mini football pitch, the club store and other entertainments 

Achievements
 Armenian Premier League
 Winner (1): 2013–14 
 Runner-up (5): 2003, 2006, 2007, 2010, 2018 

 Armenian Cup
 Winner (3): 1992, 2007, 2016 
 Runner-up (6): 2003, 2004, 2008, 2009, 2010, 2021–22

 Armenian Supercup
 Winner (1): 2014 
 Runner-up (5): 2004, 2007, 2009, 2010, 2016 

Current squad

Out on loan

Personnel

Technical staff

Management

Urartu-2

FC Banants' reserve squad play as FC Banants-2 in the Armenian First League  They play their home games at the training field with artificial turf of the Urartu Training Centre 

Managerial history
 Varuzhan Sukiasyan (1992–94)
 Poghos Galstyan (July 1, 1996 – June 30, 1998)
 Oganes Zanazanyan (2001–05)
 Ashot Barseghyan (2005–06)
 Nikolay Kiselyov (2006–07)
 Jan Poštulka (2007)
 Nikolay Kostov (July 1, 2007 – April 8, 2008)
 Nedelcho Matushev (April 8, 2008 – June 30, 2008)
 Kim Splidsboel (2008)
 Armen Gyulbudaghyants (Jan 1, 2009 – Dec 1, 2009)
 Ashot Barseghyan (interim) (2009)
 Stevica Kuzmanovski (Jan 1, 2010 – Dec 31, 2010)
 Rafael Nazaryan (Jan 1, 2011 – Jan 15, 2012)
 Volodymyr Pyatenko (Jan 17, 2013 – June 30, 2013)
 Zsolt Hornyák (July 1, 2013 – May 30, 2015)
 Aram Voskanyan (July 1, 2015 – Oct 11, 2015)
 Tito Ramallo (Oct 12, 2015 – Oct 3, 2016)
 Artur Voskanyan (Oct 3, 2016 – Aug 11, 2018)
 Ilshat Faizulin (Aug 12, 2018 –Nov 24, 2019)
 Aleksandr Grigoryan (Nov 25, 2019 –Mar 10, 2021)
 Robert Arzumanyan (10 March 2021–24 June 2022)
 Dmitri Gunko (27 June 2022–)

References

External links
 Official website 
 Banants at Weltfussball de  

 
Urartu
Urartu
Urartu
Urartu
Context 2: Sir Simon William English  (born 30 December 1961) is a New Zealand former National Party politician who served as the 39th prime minister of New Zealand from 2016 to 2017  He had previously served as the 17th deputy prime minister of New Zealand and minister of finance from 2008 to 2016 under John Key and the Fifth National Government 

A farmer and public servant before entering politics, English was elected to the New Zealand Parliament in  as the National Party's candidate in the Wallace electorate  He was elevated to Cabinet in 1996 and in 1999 was made minister of finance, although he served for less than a year due to his party's loss at the 1999 general election  In October 2001, English replaced Jenny Shipley as the leader of the National Party (and consequently as Leader of the Opposition)  He led the party to its worst defeat at the 2002 general election, and as a consequence, in October 2003 he was replaced as leader by Don Brash 

In November 2006, after Brash's resignation, English became deputy leader under John Key  After National's victory at the 2008 general election, he became deputy prime minister and was also made minister of finance for the second time  Under English's direction New Zealand's economy maintained steady growth during National's three terms of government  He became a list-only MP after stepping down as an electorate MP at the 2014 general election 

John Key resigned as leader of the National Party and prime minister in December 2016  English won the resulting leadership election unopposed and was sworn in as prime minister on 12 December 2016  His tenure was only ten months, and included a three-month election campaign  In the 2017 general election, National won the largest number of seats but fell short of a majority  The parties holding the balance of power declined to support the existing government, and English was subsequently replaced as prime minister by Jacinda Ardern, leader of the Labour Party  English initially continued on as Leader of the Opposition, but resigned as leader of the National Party on 27 February 2018 and left parliament two weeks later 

Early life
English was born on 30 December 1961 at Lumsden Maternity Centre in Lumsden  He is the eleventh of twelve children of Mervyn English and Norah (née O'Brien) English  His parents purchased Rosedale, a mixed sheep and cropping farm in Dipton, Southland from Mervyn's uncle, Vincent English, a bachelor, in 1944  English was born in the maternity unit at Lumsden 

English attended St Thomas's School in Winton, then boarded at St  Patrick's College in Upper Hutt, where he became head boy  He played in the first XV of the school's rugby team  English went on to study commerce at the University of Otago, where he was a resident at Selwyn College, and then completed an honours degree in English literature at Victoria University of Wellington 

After finishing his studies, English returned to Dipton and farmed for a few years  From 1987 to 1989, he worked in Wellington as a policy analyst for the New Zealand Treasury, at a time when the free market policies favoured by Labour's finance minister Roger Douglas (known collectively as "Rogernomics") were being implemented 

English joined the National Party in 1980, while at Victoria University  He served for a period as chairman of the Southland branch of the Young Nationals, and became a member of the Wallace electorate committee  After moving to Wellington, he served for periods on the Island Bay and Miramar electorate committees, respectively 

Fourth National Government (1990–1999)

At the 1990 general election, English stood as the National candidate in Wallace, replacing the retiring Derek Angus, and was elected with a large majority  He would hold this seat, renamed Clutha-Southland in 1996, until 2014  He and three other newly elected National MPs (Tony Ryall, Nick Smith, and Roger Sowry) were soon identified as rising stars in New Zealand politics, and at various points were dubbed the "brat pack", the "gang of four", and the "young Turks"  In his first term in parliament, English chaired a select committee into social services  He was made a parliamentary under-secretary in 1993, serving under the Minister of Health 

First period in cabinet (1996–1999)
In early 1996, English was elevated to cabinet by Prime Minister Jim Bolger, becoming the Minister for Crown Health Enterprises and Associate Minister of Education (to Wyatt Creech)  He was 34 at the time, becoming the cabinet's youngest member  After the 1996 general election, the National Party was forced into a coalition with New Zealand First to retain government  In the resulting cabinet reshuffle, English emerged as Minister of Health  However, as a condition of the coalition agreement, NZ First's Neil Kirton (a first-term MP) was made Associate Minister of Health, effectively becoming English's deputy  This arrangement was described in the press as a "shotgun marriage", and there were frequent differences of opinion between the two ministers  After their relationship became unworkable, Kirton was sacked from the role in August 1997, with the agreement of NZ First leader Winston Peters 

As Minister of Health, English was responsible for continuing the reforms to the public health system that National had begun after the 1990 general election  The reforms were unpopular, and health was perceived as one of the government's weaknesses, with the health portfolio consequently being viewed as a challenge  English believed that the unpopularity of the reforms was in part due to a failure in messaging, and encouraged his National colleagues to avoid bureaucratic and money-focused language (such as references to "balance sheets" and "user charges") and instead talk about the improvements to services the government's reforms would bring  He also rejected the idea that public hospitals could be run as commercial enterprises, a view which some of his colleagues had previously promoted 

By early 1997, as dissatisfaction with Bolger's leadership began to grow, English was being touted as a potential successor, along with Jenny Shipley and Doug Graham  His age (35) was viewed as the main impediment to a successful leadership run  National's leadership troubles were resolved in December 1997, when Bolger resigned and Shipley was elected to the leadership unopposed  English had been a supporter of Bolger as leader, but Shipley reappointed him Minister of Health in her new cabinet 

English was promoted to Minister of Finance in a reshuffle in January 1999, a position which was at the time subordinate to the Treasurer, Bill Birch  After a few months, the pair switched positions as part of Birch's transition to retirement, with English assuming the senior portfolio  In early interviews, he emphasised his wish to be seen as a pragmatist rather than an ideologue, and said that the initiatives of some of his predecessors (Roger Douglas's "Rogernomics" and Ruth Richardson's "Ruthanasia") had focused on "fruitless, theoretical debates" when "people just want to see problems solved" 

Opposition (1999–2008)

After the National Party lost the 1999 election to Helen Clark's Labour Party, English continued on in the shadow cabinet as National's spokesperson for finance  He was elected deputy leader of the party in February 2001, following the resignation of Wyatt Creech, with Gerry Brownlee being his unsuccessful opponent 

Leader of the Opposition
In October 2001, after months of speculation, Jenny Shipley resigned as leader of the National Party after being told she no longer had the support of the party caucus  English was elected as her replacement unopposed (with Roger Sowry as his deputy), and consequently became Leader of the Opposition  However, he did not openly organise against Shipley, and according to The Southland Times "there was almost an element of 'aw, shucks, I'll do it then' about Mr English's ascension" 

Aged 39 when he was elected, English became the second-youngest leader in the National Party's history, after Jim McLay (who was 38 when elected in 1984)  He also became only the third Southlander to lead a major New Zealand political party, after Joseph Ward and Adam Hamilton  However, English failed to improve the party's performance  In the 2002 election, National suffered its worst electoral defeat ever, gaining barely more than twenty percent of the vote  English described it as "the worst day of my political life"  Both party insiders and the general public were split as to how much to blame him for the loss, but most of the party believed that English would be able to rebuild National's support 

By late 2003, however, National's performance in opinion polls remained poor  The party had briefly increased its popularity in the year following the election, but by October its support had fallen to levels only slightly better than what it achieved in the last ballot  English also appeared in a boxing match for a charity against entertainer Ted Clarke  This did not boost his polling or that of the National party either, with suggestions that it devalued his image as a serious politician  Don Brash, former governor of the Reserve Bank and a relative newcomer to politics, began to build up support to replace English  On 28 October, Brash gained sufficient backing in Caucus to defeat English in a leadership contest 

Shadow cabinet roles and deputy leader
On 2 November 2003, when Brash changed responsibilities for certain MPs, English became National's spokesman for education, ranked at fifth place in the party's parliamentary hierarchy  He remained in parliament after the 2005 election  In his new shadow education portfolio, English performed strongly, and remained a party favourite despite his election defeat as leader in 2002, eventually being returned to the finance portfolio in August 2004 as deputy spokesman (while still retaining responsibility for education) 

In November 2006, Brash resigned as leader  English was considered as a potential replacement leader (running against John Key) or deputy leader (against incumbent Gerry Brownlee) in the ensuing leadership election  However, a contest was avoided when the MPs agreed a Key/English ticket would run unopposed in a display of party unity  English took over the deputy leadership and the finance portfolio in the Key shadow cabinet 

Fifth National Government (2008–2017)

Deputy Prime Minister and Minister of Finance (2008–2016)

At the 2008 election, English was re-elected by his electorate, winning by a margin of about 15,500 votes  He became Deputy Prime Minister of New Zealand and Minister of Finance in the fifth National Government, being sworn into office on 19 November 2008 and continued to serve in those roles until becoming Prime Minister on 12 December 2014  He was also made Minister of Infrastructure in National's first term of government and Minister responsible for Housing New Zealand Corporation and minister responsible for the New Zealand flag consideration process in its third 

He was comfortably re-elected in Clutha-Southland in the 2011 election but opted to run as a party-list candidate in 2014  

The pairing of John Key as leader of the National Party and English as his deputy has been compared to that of Bob Hawke and Paul Keating (in Australia) and Tony Blair and Gordon Brown (in the UK) 

English acceded to the role of Finance Minister in the continuing wake of the financial crisis  In response to New Zealand's rising debt, English made budget deficit-reduction his main priority  His first budget outlined three focuses in New Zealand's financial recovery: "improving the business environment and removing roadblocks to growth; investment in productive infrastructure; and improving the way government works"  One of his first acts was creating the National Infrastructure Unit, charged with formulating a plan for infrastructure projects and investments  He commissioned a government-wide spending review, with an aim to reducing government expenditure—with the exceptions of a two-year stimulus package and long-term increases on infrastructure spending 

In April 2011, the Opposition criticised English for suggesting that New Zealand businesses could use New Zealand's low wages to help it compete with Australia  The National Government campaigned for re-election in 2011 on its economic record  The Government boasted growth for five consecutive quarters up to mid-2010, totalling 1 6% of real GDP 

Strong growth resulted in a surplus of $473 million for the 2015/16 financial year, projected to rise to $8 5 billion by 2020/21  In his 2016 Economic and Fiscal Update address, English stated that reducing debt and tackling the costs of the 2016 Kaikōura earthquake were higher priorities than reducing rates of tax 

Allowances issue
In 2009, the media, including TVNZ and TV3 revealed that English was receiving about NZ$900 a week as part of a living allowance for ministers, to live in his own NZ$1 2 million Wellington home  At the time, English also received $276,200 in his annual salary as Deputy Prime Minister  It was also revealed other ministers with homes in the capital city were also claiming accommodation allowances  On 3 August 2009, Prime Minister John Key started a review of the housing allowances claimed by cabinet ministers  English subsequently paid back $12,000 and only claimed about $24,000 a year in living allowances  The Auditor-General's office said in September 2009 that they were making "preliminary enquiries" into parliamentary housing expenses in response to a letter of complaint from Progressive party leader Jim Anderton  Two days later English stated that he would no longer take up any housing allowance and had paid back all the allowance he had received since the November 2008 election 

Prime Minister (2016–2017)

John Key resigned on 12 December, and endorsed English as his successor in the resulting leadership election  Following the drop-out of both Judith Collins and Jonathan Coleman from the leadership election, English was sworn in as the 39th Prime Minister of New Zealand on 12 December 2016 

English appointed his first cabinet on 18 December  In a reshuffle, he appointed Steven Joyce to succeed him as Finance Minister, while most ministerial portfolios remained the same 

In February 2017, English did not attend Waitangi Day commemorations at the historic treaty grounds, reportedly in response to the Ngāpuhi iwi's decision to stop the Prime Minister from speaking at the marae  Ngāpuhi have protested the Government's negotiation of the Trans Pacific Partnership Agreement (TPPA), which the iwi believe infringes upon Māori sovereignty, and thus does not adhere to the Treaty of Waitangi  English had been invited to attend in an official capacity; his non-attendance was criticised by a Ngāpuhi elder and Opposition leader Andrew Little 

In his first overseas trip as Prime Minister, English travelled to Europe to discuss trade ties, including a prospective New Zealand–European Union free trade agreement  He first travelled to London on 13 January 2017 to meet British Prime Minister Theresa May  Discussing trade relations, English said the two nations were "natural partners" and would "continue to forge ties" after the UK's withdrawal from the EU  He also arranged to meet with London Mayor Sadiq Khan, Belgian Prime Minister Charles Michel and German Chancellor Angela Merkel   In a meeting with Merkel, English received crucial backing from Germany for a trade deal with the EU  On 16 January, English stated that his government would continue to promote TPPA, despite the United States' decision to withdraw from the agreement  He explained that Southeast Asian countries would now be treated as a priority in negotiations—he also asserted that the United States was ceding influence to China by its rejection of the trade pact 

At a press conference at the Beehive on 1 February 2017, English announced that the 2017 general election would be held on 23 September  The Prime Minister later confirmed that his party would approach ACT, United Future and the Māori Party if confidence and supply agreements were required to form a government following the election  In his second cabinet reshuffle on 24 April, English appointed Gerry Brownlee as his new Foreign Affairs Minister; he also promoted Nikki Kaye to the portfolio of Education Minister, and moved Mark Mitchell into the cabinet to become Defence Minister  The reshuffle was perceived as an election preparation 

On 13 February 2017, English welcomed Australian Prime Minister Malcolm Turnbull to Wellington  The two leaders reaffirmed their shared trade agenda, and discussed changes to the Australian citizenship pathway which will affect permanent residents originating from New Zealand 

On 19 June, it was reported that Todd Barclay, who succeeded English as MP for Clutha-Southland, had clandestinely recorded one of his employee's conversations the previous year, and that John Key's leaders' budget was used to pay a confidential settlement after the employee resigned  English admitted that he had been aware of the illegal recording and the settlement, and thus implicated in the scandal 

During the 2017 National campaign launch, English introduced a $379 million social investment package including digital learning academies for high school students, more resources for mathematics, and boosting support for teaching second languages in schools, and maintaining National Standards in the school curriculum  Prime Minister English also sought to defend National's financial management and economic track record and claimed that the opposition Labour Party would raise taxes  Early opinion polling had forecast a poor showing in the election for the Labour Party, but in early August 37-year-old Jacinda Ardern took over as Labour leader and seemingly energised younger voters 

At the 2017 general election, National won the largest share of the party vote (44 4%) and the largest number of seats (56) in the House Representatives  However, National lacked enough seats to govern alone due to two of the party's support partners, the Māori Party and United Future, losing their parliamentary seats  In response, English stated that the party would be entering into talks to form a coalition with New Zealand First  Following talks with the two largest parties, New Zealand First entered a coalition arrangement with the Labour Party  English was succeeded as prime minister by Jacinda Ardern on 26 October 

Opposition (2017–2018)

Leader of the Opposition
English was re-elected as National Party leader on 24 October 2017  At the time of his re-election, English announced his intention to stay on as leader until the next general election  On 13 February 2018, however, he stood down as National Party leader due to personal reasons, and instructed the party to put into motion the processes to elect a new leader  He also retired from Parliament  English's resignation followed weeks of speculation that he would step aside for a new leader  On 27 February, he was succeeded as party leader by Simon Bridges as the result of the leadership election held that day 

Post-premiership 
In 2018, English joined the board of Australian conglomerate, Wesfarmers  English serves in Chairmanships of Mount Cook Alpine Salmon, Impact Lab Ltd and Manawanui Support Ltd  He is also a director of The Instillery, Centre for Independent Studies and The Todd Corporation Limited, and is a member of the Impact Advisory Group of Macquarie Infrastructure and Real Assets 

Political and social views

English is regarded as more socially conservative than his predecessor, John Key  He has stated his opposition to voluntary euthanasia and physician-assisted suicide, same-sex civil unions, and the decriminalisation of prostitution  As Prime Minister he opposed any "liberalisation" of abortion law 

In 2004, English voted against a bill to establish civil unions for both same-sex and opposite-sex couples  In 2005, he voted for the Marriage (Gender Clarification) Amendment Bill, which would have amended the Marriage Act to define marriage as only between a man and a woman  English voted against the Marriage (Definition of Marriage) Amendment Bill, a bill that legalised same-sex marriage in New Zealand  However, in December 2016 he stated, "I'd probably vote differently now on the gay marriage issue  I don't think that gay marriage is a threat to anyone else's marriage" 

In 2009, English voted against the Misuse of Drugs (Medicinal Cannabis) Amendment Bill, a bill aimed at amending the Misuse of Drugs Act so that cannabis could be used for medical purposes 

Personal life 
English met his future wife, Mary Scanlon, at university  She was studying medicine at the time, and became a general practitioner  Both her parents were immigrants, her father being Samoan and her mother Italian, born on the island of Stromboli  They have six children: a daughter and five sons 

English is a practising Roman Catholic, but has stated that he considers his religious beliefs personal and thus separate from politics 

In June 2002, English took part in TV3's Fight For Life, a celebrity boxing fundraiser to raise money for the Yellow Ribbon anti-youth-suicide campaign, influenced by the death of a teenage nephew in 1997  He lost a split decision to former university colleague Ted Clarke 

Honours
In the 2018 Queen's Birthday Honours, English was appointed a Knight Companion of the New Zealand Order of Merit, for services of over 27 years to the State 



ground_truth: FC Banants.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: When did the 2017 general election be held?

Read the following text and answer briefly.

Context 3: Paper Info

Title: On the Role of Emergent Communication for Social Learning in Multi-Agent Reinforcement Learning
Publish Date: Unkown
Author List: Seth Karten, Siva Kailas, Huao Li, Katia Sycara

Figure

Figure1 By using contrastive learning, our method seeks similar representations between the state-message pair and future states while creating dissimilar representations with random states Thus satisfying the utility objective of the information bottleneck The depicted agents are blind and cannot see other cars 
Figure 2 An example of two possible classes, person and horse, from a single observation in the Pascal VOC game 
Figure 3  Blind Traffic Junction Left: Our method uses compositional complexity and contrastive utility to outperform other baselines in terms of performance and sample complexity The legend provides the mean ± variance of the best performance Right: Top: success, contrastive, and complexity losses for our method Right, Bottom: success, autoencoder loss for ae-comm with supervised pretraining 
Figure 4  Pascal VOC Game Representing compositional concepts from raw pixel data in images to communicate multiple concepts within a single image Our method significantly outperforms ae-comm and no-comm due to our framework being able to learn composable, independent concepts 
Figure 5  Blind Traffic Junction Social shadowing enables significantly lower sample complexity when compared to traditional online MARL 
Beta ablation: Messages are naturally sparse in bits due to the complexity loss Redundancy measures the capacity for a bijection between the size of the set of unique tokens and the enumerated observations and intents Min redundancy is 1 0 (a bijection) Lower is better 

abstract

Explicit communication among humans is key to coordinating and learning  Social learning, which uses cues from experts, can greatly benefit from the usage of explicit communication to align heterogeneous policies, reduce sample complexity, and solve partially observable tasks  Emergent communication, a type of explicit communication, studies the creation of an artificial language to encode a high task-utility message directly from data 
However, in most cases, emergent communication sends insufficiently compressed messages with little or null information, which also may not be understandable to a third-party listener  This paper proposes an unsupervised method based on the information bottleneck to capture both referential complexity and task-specific utility to adequately explore sparse social communication scenarios in multi-agent reinforcement learning (MARL) 
We show that our model is able to i) develop a natural-language-inspired lexicon of messages that is independently composed of a set of emergent concepts, which span the observations and intents with minimal bits, ii) develop communication to align the action policies of heterogeneous agents with dissimilar feature models, and iii) learn a communication policy from watching an expert's action policy, which we term 'social shadowing' 

INTRODUCTION

Social learning agents analyze cues from direct observation of other agents (novice or expert) in the same environment to learn an action policy from others  However, observing expert actions may not be sufficient to coordinate with other agents  Rather, by learning to communicate, agents can better model the intent of other agents, leading to better coordination 
In humans, explicit communication for coordination assumes a common communication substrate to convey abstract concepts and beliefs directly , which may not be available for new partners  To align complex beliefs, heterogeneous agents must learn a message policy that translates from one theory of mind to another to synchronize coordination 
Especially when there is complex information to process and share, new agent partners need to learn to communicate to work with other agents  Emergent communication studies the creation of artificial language  Often phrased as a Lewis game, speakers and listeners learn a set of tokens to communicate complex observations  
However, in multi-agent reinforcement learning (MARL), agents suffer from partial observability and non-stationarity (due to unaligned value functions) , which aims to be solved with decentralized learning through communication  In the MARL setup, agents, as speakers and listeners, learn a set of tokens to communicate observations, intentions, coordination, or other experiences which help facilitate solving tasks  
Agents learn to communicate effectively through a backpropagation signal from their task performance   This has been found useful for applications in human-agent teaming , multirobot navigation , and coordination in complex games such as StarCraft II   Communication quality has been shown to have a strong relationship with task performance , leading to a multitude of work attempting to increase the representational capacity by decreasing the convergence rates  
Yet these methods still create degenerate communication protocols , which are uninterpretable due to joined concepts or null (lack of) information, which causes performance degradation  In this work, we investigate the challenges of learning a arXiv:2302 14276v1 LG] 28 Feb 2023 messaging lexicon to prepare emergent communication for social learning (EC4SL) scenarios 
We study the following hypotheses: H1) EC4SL will learn faster through structured concepts in messages leading to higher-quality solutions, H2) EC4SL aligns the policies of expert heterogeneous agents, and H3) EC4SL enables social shadowing, where an agent learns a communication policy while only observing an expert agent's action policy 
By learning a communication policy, the agent is encouraged to develop a more structured understanding of intent, leading to better coordination  The setting is very realistic among humans and many computer vision and RL frameworks may develop rich feature spaces for a specific solo task, but have not yet interacted with other agents, which may lead to failure without alignment 
We enable a compositional emergent communication paradigm, which exhibits clustering and informativeness properties  We show theoretically and through empirical results that compositional language enables independence properties among tokens with respect to referential information  Additionally, when combined with contrastive learning, our method outperforms competing methods that only ground communication on referential information 
We show that contrastive learning is an optimal critic for communication, reducing sample complexity for the unsupervised emergent communication objective  In addition to the more human-like format, compositional communication is able to create variable-length messages, meaning that we are not limited to sending insufficiently compressed messages with little information, increasing the quality of each communication 
In order to test our hypotheses, we show the utility of our method in multi-agent settings with a focus on teams of agents, high-dimensional pixel data, and expansions to heterogeneous teams of agents of varying skill levels  Social learning requires agents to explore to observe and learn from expert cues 
We interpolate between this form of social learning and imitation learning, which learns action policies directly from examples  We introduce a 'social shadowing' learning approach where we use first-person observations, rather than third-person observations, to encourage the novice to learn latently or conceptually how to communicate and develop an understanding of intent for better coordination 
The social shadowing episodes are alternated with traditional MARL during training  Contrastive learning, which works best with positive examples, is apt for social shadowing  Originally derived to enable lower complexity emergent lexicons, we find that the contrastive learning objective is apt for agents to develop internal models and relationships of the task through social shadowing 
The idea is to enable a shared emergent communication substrate (with minimal bandwidth) to enable future coordi-nation with novel partners  Our contributions are deriving an optimal critic for a communication policy and showing that the information bottleneck helps extend communication to social learning scenarios 
In real-world tasks such as autonomous driving or robotics, humans do not necessarily learn from scratch  Rather they explore with conceptually guided information from expert mentors  In particular, having structured emergent messages reduces sample complexity, and contrastive learning can help novice agents learn from experts 
Emergent communication can also align heterogeneous agents, a social task that has not been previously studied 

Multi-Agent Signaling

Implicit communication conveys information to other agents that is not intentionally communicated   Implicit signaling conveys information to other agents based on one's observable physical position   Implicit signaling may be a form of implicit communication such as through social cues or explicit communication such as encoded into the MDP through "cheap talk"  
Unlike implicit signaling, explicit signaling is a form of positive signaling that seeks to directly influence the behavior of other agents in the hopes that the new information will lead to active listening  Multi-agent emergent communication is a type of explicit signaling which deliberately shares information 
Symbolic communication, a subset of explicit communication, seeks to send a subset of pre-defined messages  However, these symbols must be defined by an expert and do not scale to particularly complex observations and a large number of agents  Emergent communication aims to directly influence other agents with a learned subset of information, which allows for scalability and interpretability by new agents 

Emergent Communication

Several methodologies currently exist to increase the informativeness of emergent communication  With discrete and clustered continuous communication, the number of observed distinct communication tokens is far below the number permissible   As an attempt to increase the emergent "vocabulary" and decrease the data required to converge to an informative communication "language", work has added a bias loss to emit distinct tokens in different situations  
More recent work has found that the sample efficiency can be further improved by grounding communication in observation space with a supervised reconstruction loss   Information-maximizing autoencoders aim to maximize the state reconstruction accuracy for each agent  How-ever, grounding communication in observations has been found to easily satisfy these input-based objectives while still requiring a myriad more samples to explore to find a task-specific communication space  
Thus, it is necessary to use task-specific information to communicate informatively  This will enable learned compression for task completion rather than pure compression for input recovery  Other work aims to use the information bottleneck to decrease the entropy of messages   In our work, we use contrastive learning to increase representation similarity with future goals, which we show optimally optimizes the Q-function for messages 

Natural Language Inspiration

The properties of the tokens in emergent communication directly affect their informative ability  As a baseline, continuous communication tokens can represent maximum information but lack human-interpretable properties  Discrete 1-hot (binary vector) tokens allow for a finite vocabulary, but each token contains the same magnitude of information, with equal orthogonal distance to each other token 
Similar to word embeddings in natural language, discrete prototypes are an effort to cluster similar information together from continuous vectors   Building on the continuous word embedding properties, VQ-VIB , an information-theoretic observation grounding based on VQ-VAE properties , uses variational properties to provide word embedding properties for continuous emergent tokens 
Like discrete prototypes, they exhibit a clustering property based on similar information but are more informative  However, each of these message types determines a single token for communication  Tokens are stringed together to create emergent "sentences" 

Preliminaries

We formulate our setup as a decentralized, partially observable Markov Decision Process with communication (Dec-POMDP-Comm)  Formally, our problem is defined by the tuple, S, A, M, T , R, O, Ω, γ   We define S as the set of states, A i , i ∈ [1, N ] as the set of actions, which includes task-specific actions, and M i as the set of communications for N agents 
T is the transition between states due to the multi-agent joint action space T : S × A 1 ,    , A N → S  Ω defines the set of observations in our partially observable setting  Partial observability requires communication to complete the tasks successfully  O i : M 1 ,    , M N × Ŝ → Ω maps the communications and local state, Ŝ, to a distribution of observations for each agent 
R defines the reward function and γ defines the discount factor 

Architecture

The policy network is defined by three stages: Observation Encoding, Communication, and Action Decoding  The best observation encoding and action decoding architecture is task-dependent, i e , using multi-layer perceptrons (MLPs), CNNs , GRUs , or transformer layers are best suited to different inputs 
The encoder transforms observation and any sequence or memory information into an encoding H  The on-policy reinforcement learning training uses RE-INFORCE or a decentralized version of MAPPO as specified by our experiments  Our work focuses on the communication stage, which can be divided into three substages: message encoding, message passing (often considered sparse communication), and message decoding 
We use the message passing from   For message decoding, we build on a multiheaded attention framework, which allows an agent to learn which messages are most important   Our compositional communication framework defines the message encoding, as described in section 4 

Objective

Mutual information, denoted as I(X; Y ), looks to measure the relationship between random variables, which is often measured through Kullback-Leibler divergence , I(X; Y ) = D KL (p(x, y)||p(x) ⊗ p(y))  The message encoding substage can be defined as an information bottleneck problem, which defines a tradeoff between the complexity of information (compression, I(X, X)) and the preserved relevant information (utility, I( X, Y )) 
The deep variational information bottleneck defines a trade-off between preserving useful information and compression   We assume that our observation and memory/sequence encoder provides an optimal representation H i suitable for sharing relevant observation and intent/coordination information  We hope to recover a representation Y i , which contains the sufficient desired outputs 
In our scenario, the information bottleneck is a trade-off between the complexity of information I(H i ; M i ) (representing the encoded information exactly) and representing the relevant information I(M j =i ; Y i ), which is signaled from our contrastive objective  In our setup, the relevant information flows from other agents through communication, signaling a combination of the information bottleneck and a Lewis game 
We additionally promote complexity through our compositional independence objective, This is formulated by the following Lagrangian, where the bounds on mutual information Î are defined in equations 1, 2, and 10  Overall, our objective is,

Complexity through Compositional Communication

We aim to satisfy the complexity objective, I(H i , M i ), through compositional communication  In order to induce complexity in our communication, we want the messages to be as non-random as possible  That is, informative with respect to the input hidden state h  In addition, we want each token within the message to share as little information as possible with the preceding tokens 
Thus, each additional token adds only informative content  Each token has a fixed length in bits W   The total sequence is limited by a fixed limit, L l W l ≤ S, of S bits and a total of L tokens  We use a variational message generation setup, which maps the encoded hidden state h to a message m; that is, we are modeling the posterior, π i m (m l |h) 
We limit the vocabulary size to K tokens, e j ∈ R D , j ∈ [1, K] ⊂ N, where each token has dimensionality D and l ∈ [1, L] ⊂ N  Each token m l is sampled from a categorical posterior distribution, 0 otherwise such that the message m l is mapped to the nearest neighbor e j   A set of these tokens makes a message m 
To satisfy the complexity objective, we want to use m i to well-represent h i and consist of independently informative m i l  

Independent Information

We derive an upper bound for the interaction information between all tokens  Proposition 4 1  For the interaction information between all tokens, the following upper bound holds: The proof is in Appendix A 1  Since we want the mutual information to be minimized in our objective, we minimize,

Input-Oriented Information

In order to induce complexity in the compositional messages, we additionally want to minimize the mutual information I(H; M ) between the composed message m and the encoded information h  We derive an upper bound on the mutual information that we use as a Lagrangian term to minimize  Proposition 4 2  For the mutual information between the composed message and encoded information, the following upper bound holds:
The proof is in Appendix A 1  Thus, we have our Lagrangian term, Conditioning on the input or observation data is a decentralized training objective 

Sequence Length

Compositional communication necessitates an adaptive limit on the total length of the sequence  Corollary 4 3  Repeat tokens, w, are redundant and can be removed  Suppose one predicts two arbitrary tokens, w k and w l   Given equation 1, it follows that there is low or near-zero mutual information between w k and w l  
A trivial issue is that the message generator will predict every available token as to follow the unique token objective  Since the tokens are imbued with input-oriented information (equation 2), the predicted tokens will be based on relevant referential details  Thus, it follows that tokens containing irrelevant information will not be chosen 
A nice optimization objective that follows from corollary 4 3 is that one can use self-supervised learning with an end-ofsequence (EOS) token to limit the variable total length of compositional message sequences  (3) Algorithm 1 Compositional Message Gen (h t ) m i ∼ N ( ĥ; µ, σ) 9: end for 10: return m

Message Generation Architecture

Now, we can define the pipeline for message generation  The idea is to create an architecture that can generate features to enable independent message tokens  We expand each compressed token into the space of the hidden state h (1-layer linear expansion) since each token has a natural embedding in R |h|  
Then, we perform attention using a softmin to help minimize similarity with previous tokens and sample the new token from a variational distribution  See algorithm 1 for complete details  During execution, we can generate messages directly due to equation 1, resolving any computation time lost from sequential compositional message generation 

Utility through Contrastive Learning

First, note that our Markov Network is as follows: H j → M j → Y i ← H i   Continue to denote i as the agent identification and j as the agent ID such that j = i  We aim to satisfy the utility objective of the information bottleneck, I(M j ; Y i ), through contrastive learning as shown in figure 1  Proposition 5 1 
Utility mutual information is lower bounded by the contrastive NCE-binary objective, The proof is in Appendix A 1  This result shows a need for gradient information to flow backward across agents along communication edge connections 

Experiments and Results

We condition on inputs, especially rich information (such as pixel data), and task-specific information  When evaluating an artificial language in MARL, we are interested in referential tasks, in which communication is required to complete the task  With regard to intent-grounded communication, we study ordinal tasks, which require coordination information between agents to complete successfully 
Thus, we consider tasks with a team of agents to foster messaging that communicates coordination information that also includes their observations  To test H1, structuring emergent messages enables lower complexity, we test our methodology and analyze the input-oriented information and utility capabilities 
Next, we analyze the ability of heterogeneous agents to understand differing communication policies (H2))  Finally, we consider the effect of social shadowing (H3), in which agents solely learn a communication policy from an expert agent's action policy  We additionally analyze the role of offline reinforcement learning for emergent communication in combination with online reinforcement learning to further learn emergent communication alongside an action policy 
We evaluate each scenario over 10 seeds 

Environments

Blind Traffic Junction We consider a benchmark that requires both referential and ordinal capabilities within a team of agents  The blind traffic junction environment requires multiple agents to navigate a junction without any observation of other agents  Rather, they only observe their own state location 
Ten agents must coordinate to traverse through the lanes without colliding into agents within their lane or in the junction



ground_truth: 23 September.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What is the main methodology used in the research?

Read the following text and answer briefly.

Context 3:  Our training uses REINFORCE   Pascal VOC Game We further evaluate the complexity of compositional communication with a Pascal VOC   This is a two-agent referential game similar to the Cifar game but requires the prediction of multiple classes 
During each episode, each agent observes a random image from the Pascal VOC dataset containing exactly two unique labels  Each agent must encode information given only the raw pixels from the original image such that the other agent can recognize the two class labels in the original image  An agent receives a reward of 0 25 per correctly chosen class label and will receive a total reward of 1 if both agents guess all labels correctly 
See figure 2  Our training uses heterogeneous agents trained with PPO (modified from MAPPO repository)  For simplicity of setup, we consider images with exactly two unique labels from a closed subset of size five labels of the original set of labels from the Pascal VOC data  Furthermore, these images must be of size 375 × 500 pixels 
Thus, the resultant dataset comprised 534 unique images from the Pascal VOC dataset 

Baselines

To evaluate our methodology, we compare our method to the following baselines: (1) no-comm, where agents do not communicate; (2) rl-comm, which uses a baseline communication method learned solely through policy loss ; (3) ae-comm, which uses an autoencoder to ground communication in input observations ; (4) VQ-VIB, which uses a variational autoencoder to ground discrete communication in input observations and a mutual information objective to ensure low entropy communication  
We provide an ablation of the loss parameter β in table 1 in the blind traffic junction scenario  When β = 0, we use our compositional message paradigm without our derived loss terms  We find that higher complexity and independence losses increase sample complexity  When β = 1, the model was unable to converge 
However, when there is no regularization loss, the model performs worse (with no guarantees about referential representation)  We attribute this to the fact that our independence criteria learns a stronger causal relationship  There are fewer spurious features that may cause an agent to take an incorrect action 
In order to understand the effect of the independent concept representation, we analyze the emergent language's capacity for redundancy  A message token m l is redundant if there exists another token m k that represents the same information  With our methodology, the emergent 'language' converges to the exact number of observations and intents required to solve the task 
With a soft discrete threshold, the independent information loss naturally converges to a discrete number of tokens in the vocabulary  Our β ablation in table 1 yields a bijection between each token in the vocabulary and the possible emergent concepts, i e , the enumerated observations and intents  Thus for β = 0 1, there is no redundancy 
Sparse Communication In corollary 4 3, we assume that there is no mutual information between tokens  In practice, the loss may only be near-zero  Our empirical results yield independence loss around 1e − 4  In table 1, the size of the messages is automatically compressed to the smallest size to represent the information 
Despite a trivially small amount of mutual information between tokens, our compositional method is able to reduce the message size in bits by 2 3x using our derived regularization, for a total of an 8x reduction in message size over non-compositional methods such as ae-comm  Since the base unit for the token is a 32-bit float, we note that each token in the message may be further compressed 
We observe that each token uses three significant digits, which may further compress tokens to 10 bits each for a total message length of 20 bits 

Communication Utility Results

Due to coordination in MARL, grounding communication in referential features is not enough  Finding the communication utility requires grounding messages in ordinal information  Overall, figure shows that our compositional, contrastive method outperforms all methods focused on solely input-oriented communication grounding 
In the blind traffic junction, our method yields a higher average task success rate and is able to achieve it with a lower sample complexity  Training with the contrastive update tends to spike to high success but not converge, often many episodes before convergence, which leaves area for training improvement 
That is, the contrastive update begins to find aligned latent spaces early in training, but it cannot adapt the methodology quickly enough to converge  The exploratory randomness of most of the early online data prevents exploitation of the high utility f + examples  This leaves further room for improvement for an adaptive contrastive loss term 
Regularization loss convergence After convergence to high task performance, the autoencoder loss increases in order to represent the coordination information  This follows directly from the information bottleneck, where there exists a tradeoff between utility and complexity  However, communication, especially referential communication, should have an overlap between utility and complexity 
Thus, we should seek to make the complexity loss more convex  Our compositional communication complexity loss does not converge before task performance convergence  While the complexity loss tends to spike in the exploratory phase, the normalized value is very small  Interestingly, the method eventually converges as the complexity loss converges below a normal- ized 0 3 
Additionally, the contrastive loss tends to decrease monotonically and converges after the task performance converges, showing a very smooth decrease  The contrastive f − loss decreases during training, which may account for success spikes prior to convergence  The method is able to converge after only a moderate decrease in the f + loss 
This implies empirical evidence that the contrastive loss is an optimal critic for messaging  See figure 3 

Heterogeneous Alignment Through Communication

In order to test the heterogeneous alignment ability of our methodology to learn higher-order concepts from highdimensional data, we analyze the performance on the Pascal VOC game  We compare our methodology against ae-comm to show that concepts should consist of independent information directly from task signal rather than compression to reconstruct inputs 
That is, we show an empirical result on pixel data to verify the premise of the information bottleneck  Our methodology significantly outperforms the observation-grounded ae-comm baseline, as demonstrated by figure 4  The ae-comm methodology, despite using autoencoders to learn observation-grounded communication, performs only slightly better than no-comm 
On the other hand, our methodology is able to outperform both baselines significantly  It is important to note that based on figure 4, our methodology is able to guess more than two of the four labels correctly across the two agents involved, while the baseline methodologies struggle to guess exactly two of thew four labels consistently 
This can be attributed to our framework being able to learn compositional concepts that are much more easily discriminated due to mutual independence 

Social Shadowing

Critics of emergent communication may point to the increased sample complexity due to the dual communication and action policy learning  In the social shadowing scenario, heterogeneous agents can learn to generate a communication policy without learning the action policy of the watched expert agents  To enable social shadowing, the agent will alternate between a batch of traditional MARL (no expert) and (1st-person) shadowing an expert agent performing the task in its trajectory 
The agent only uses the contrastive objective to update its communication policy during shadowing  In figure , the agent that performs social shadowing is able to learn the action policy with almost half the sample complexity required by the online reinforcement learning agent  Our results show that the structured latent space of the emergent communication learns socially benevolent coordination 
This tests our hypothesis that by learning communication to understand the actions of other agents, one can enable lower sample complexity coordination  Thus, it mitigates the issues of solely observing actions 

Discussion

By using our framework to better understand the intent of others, agents can learn to communicate to align policies and coordinate  Any referential-based setup can be performed with a supervised loss, as indicated by the instant satisfaction of referential objectives  Even in the Pascal VOC game, which appears to be a purely referential objective, our results show that intelligent compression is not the only objective of referential communication 
The emergent communication paradigm must enable an easy-to-discriminate space for the game  In multi-agent settings, the harder challenge is to enable coordination through communication  Using contrastive communication as an optimal critic aims to satisfy this, and has shown solid improvements  Since contrastive learning benefits from good examples, this method is even more powerful when there is access to examples from expert agents 
In this setting, the communication may be bootstrapped, since our optimal critic has examples with strong signals from the 'social shadowing' episodes  Additionally, we show that the minimization of our independence objective enables tokens that contain minimal overlapping information with other tokens 
Preventing trivial communication paradigms enables higher performance  Each of these objectives is complementary, so they are not trivially minimized during training, which is a substantial advantage over comparative baselines  Unlike prior work, this enables the benefits of training with reinforcement learning in multi-agent settings 
In addition to lower sample complexity, the mutual information regularization yields additional benefits, such as small messages, which enables the compression aspect of sparse communication  From a qualitative point of view, the independent information also yields discrete emergent concepts, which can be further made human-interpretable by a post-hoc analysis  
This is a step towards white-box machine learning in multi-agent settings  The interpretability of this learned white-box method could be useful in human-agent teaming as indicated by prior work   The work here will enable further results in decision-making from high-dimensional data with emergent concepts 
The social scenarios described are a step towards enabling a zero-shot communication policy  This work will serve as future inspiration for using emergent communication to enable ad-hoc teaming with both agents and humans 

Appendix

A 1  Proofs Proposition 4 1 For the interaction information between all tokens, the following upper bound holds: Proof  Starting with the independent information objective, we want to minimize the interaction information, which defines the conditional mutual information between each token and, Let π i m (m l |h) be a variational approximation of p(m l |h), which is defined by our message encoder network 
Given that each token should provide unique information, we assume independence between m l   Thus, it follows that our compositional message is a vector, m = [m 1 ,       , m L ], and is jointly Gaussian  Moreover, we can define q( m|h) as a variational approximation to p(m|h) = p(m 1 ;       , m L |h) 
We can model q with a network layer and define its loss as || m − m|| 2   Thus, transforming equation 4 into variational form, we have, it follows that q( m|h) log q( m|h)d m ≥ q( m|h) log Thus, we can bound our interaction information, Proposition 4 2 For the mutual information between the composed message and encoded information, the following upper bound holds:
Proof  By definition of mutual information between the composed messages M and the encoded observations H, we have, Substituting q( m|h) for p( m|h), the same KL Divergence identity, and defining a Gaussian approximation z( m) of the marginal distribution p( m), it follows that, In expectation of equation 1, we have,
This implies that, for m = [m 1 ,       , m L ], there is probabilistic independence between m j , m k , j = k  Thus, expanding, it follows that, where z(m l ) is a standard Gaussian  Proposition 5 1  Utility mutual information is lower bounded by the contrastive NCE-binary objective, Proof  We suppress the reliance on h since this is directly passed through 
By definition of mutual information, we have, Our network model learns π R + (y|m) from rolled-out trajectories, R + , using our policy  The prior of our network state, π R − (y), can be modeled from rolling out a random trajectory, R−  Unfortunately, it is intractable to model π R + (y|m) and π R − (y) directly during iterative learning, but we can sample y + ∼ π R + (y|m) and y − ∼ π R − (y) directly from our network during training 
It has been shown that log p(y|m) provides a bound on mutual information , with the expectation over l p(m l , y l )  However, we need a tractable understanding of the information Y   In the information bottleneck, Y represents the desired outcome  In our setup, y is coordination information that helps create the desired output, such as any action a −  
This implies, y =⇒ a −   Since the transition is known, it follows that a − =⇒ s − f , a random future state  Thus, we have, π This is similar to the proof for lemma A 5, but requires assumptions on messages m from the emergent language  We note that when m is random, the case defaults to lemma A 5  Thus, we assume we have at least input-oriented information in m given sufficiently satisfying equation 2  Given a sufficient emergent language, it follows that y =⇒ a + , where a + is an intention action based on m 
Similarly, since the transition is known, a + =⇒ s + f , a desired goal state along the trajectory  Thus, we have, π R + (y|m) = p(s = s + f |y, m)  Recall the following (as shown in ), which we have adapted to our communication objective, Proposition A 3 (rewards → probabilities)  The Q-function for the goal-conditioned reward function r g (s t , m t ) = (1 − γ)p(s = s g |y t ) is equivalent to the probability of state s g under the discounted state occupancy measure:
and Lemma A 4  The critic function that optimizes equation 8 is a Q-function for the goal-conditioned reward function up to a multiplicative constant 1 The critic function f (s, m, s f ) = y enc(s f ) represents the similarity between the encoding y = enc(s, m) and the encoding of the future rollout s f  
Given lemmas A 5 A 6 A 8 and proposition A 7, it follows that equation 8 is the NCE-binary (InfoMAX ) objective, Î(M j , Y i ) = log σ(f (s, m, s + f )) + log 1 − σ(f (s, m, s − f )) which lower bounds the mutual information, I(M j , Y i ) ≥ Î(M j , Y i )  The critic function is unbounded, so we constrain it to [0, 1] with the sigmoid function, σ( * ) 
We suppress the reliance on h since this is directly passed through  By definition of mutual information, we have, Our network model learns π R + (y|m) from rolled-out trajectories, R + , using our policy  The prior of our network state, π R − (y), can be modeled from rolling out a random trajectory, R− 
Unfortunately, it is intractable to model π R + (y|m) and π R − (y) directly during iterative learning, but we can sample y + ∼ π R + (y|m) and y − ∼ π R − (y) directly from our network during training  It has been shown that log p(y|m) provides a bound on mutual information , with the expectation over l p(m l , y l ) 
However, we need a tractable understanding of the information Y   Lemma A 5  π R − (y) = p(s = s − f |y)  In the information bottleneck, Y represents the desired outcome  In our setup, y is coordination information that helps create the desired output, such as any action a −   This implies, y =⇒ a −   Since the transition is known, it follows that a − =⇒ s − f , a random future state 
Thus, we have, π R − (y) = p(s = s − f |y)  Lemma A 6  π R + (y|m) = p(s = s + f |y, m)  This is similar to the proof for lemma A 5, but requires assumptions on messages m from the emergent language  We note that when m is random, the case defaults to lemma A 5  Thus, we assume we have at least input-oriented information in m given sufficiently satisfying equation 2  Given a sufficient emergent language, it follows that y =⇒ a + , where a + is an intention action based on m 
Similarly, since the transition is known, a + =⇒ s + f , a desired goal state along the trajectory  Thus, we have, π R + (y|m) = p(s = s + f |y, m)  Recall the following (as shown in ), which we have adapted to our communication objective, Proposition A 7 (rewards → probabilities)  The Q-function for the goal-conditioned reward function r g (s t , m t ) = (1 − γ)p(s = s g |y t ) is equivalent to the probability of state s g under the discounted state occupancy measure:
and Lemma A 8  The critic function that optimizes equation 8 is a Q-function for the goal-conditioned reward function up to a multiplicative constant 1 p(s f ) : exp(f * (s, m, s f ) = 1 p(s f ) Q π s f (s, m)  The critic function f (s, m, s f ) = y enc(s f ) represents the similarity between the encoding y = enc(s, m) and the encoding of the future rollout s f  
Given lemmas A 5 A 6 A 8 and proposition A 7, it follows that equation 8 is the NCE-binary (InfoMAX ) objective, which lower bounds the mutual information, I(M j , Y i ) ≥ Î(M j , Y i )  The critic function is unbounded, so we constrain it to [0, 1] with the sigmoid function, σ( * ) 



ground_truth: An unsupervised method based on the information bottleneck and contrastive learning.