Assayag

Part B1REACH

ERC Advanced Grant 2019
Research proposal [Part B1]

Raising co-creativity in cyber-human Musicianship
REACH
Name of the Principal Investigator (PI): Gérard Assayag
Name of the PI's host institution for the project: Institut de Recherche et de Coordination Acoustique /Musique (IRCAM)
Proposal duration in months: 60
Raising co-creativity in cyber-human musicianship
Digital societies will see the development of deep interweaving between human creativity and autonomous
information-processing capabilities of virtual and physical objects, that turn joint human-machine activity intoa mixed reality with symbiotic interactions. In the cultural or industrial domains, co-creativity between humans
and machines will bring about the emergence of distributed information structures of a new kind that will
profoundly impact individual and collective human development. With technologies allowing one to extract
semantic features from physical and human signals, combined with generative learning of high-level
representations, we are beginning to unveil the increasing complexity of cooperation, synergy or conflicts
inherent to cyber-human networks. These hybrid micro-societies will need renewed frames of thoughts andactions in order to be fully understood and managed. To this end the REACH project aims at understanding,modeling, and developing co-creativity between humans and machines in the musical domain throughimprovised interactions allowing musicians of any level of training to develop their skills and expand theirindividual and social creative potential. Indeed, improvisation is at the very heart of all human interactions,and music is a fertile ground for developing models and tools of creativity that can be generalized to otheractivities, as in music the constraints are among the strongest to conduct cooperative or competitive behaviors
that come together into highly integrated courses of actions. REACH will study shared musicianship occurringat the intersection of the physical, human and digital spheres, as an archetype of distributed intelligence, andwill produce powerful models and tools as vehicles to better understand and foster human creativity in a world
where it becomes more and more intertwined with computation.

1

AssayagPart B1

REACH

Section a: Extended Synopsis of the scientific proposal (max. 5 pages)

Project outline
The research objective of this inter-disciplinary project is to model and enhance co-creativity as it arises in
improvised musical interactions between human and artificial agents, in a spectrum of practices spanning frominteracting with software agents to mixed reality involving instrumental physicality and embodiment. Suchcreative interaction strongly involves co-improvisation, as a mixture of more or less predictable events,
reactive and planned behaviors, discovery and action phases, states of volition or idleness. Improvisation isthus at the core of this project and indeed a fundamental constituent of co-creative musicianship, as well as afascinating anthropological lever to human interactions in general.
The outline of the project unfolds as follows:ü Understanding, modelling, implementing music generativity and improvised interaction as a general
template for symbiotic interaction between humans and digital systems (cyber-human systems)
ü Creating the scientific and technological conditions for mixed reality musical systems, based on theinterrelation of creative agents and active control in physical systems.
ü Achieving distributed co-creativity through complex temporal adaptation of creative agents in live cyberhuman systems, articulated to field experiment in musical social sciences.
These three directions are already innovative and ambitious in their own right. Setting up a synergy between
them, in order to foster understanding of co-creativity between natural and artificial beings on a sociallysignificant scale, is an entirely novel initiative. It is the first organized interdisciplinary attempt at decipheringand shaping cyber-human co-creativity, an emerging field of study obviously called to considerable
developments in the digital era. Revealing the mechanisms of co-creativity in music will also be a remarkableprojector on creativity in general, as music is one of the most highly organized, interactive and complex humanactivity — at the same time an abstract, sensitive and physical one, yet profoundly shaped by communicationand improvisation, and as such a powerful metaphor of human creative interactions.
We will put in place a research ecosystem allowing us to answer two central questions:ü How to augment the digital agents capacities with enhanced computational creativity as well as withcyber-physical extensions, so they can develop convincing interactions with humans
ü How to augment the human capacities by expanding their individual and social creative potential, throughnovel collaborative strategies and mixed reality, so they can naturally immerse themselves in complex
schemes involving digital intelligence.
In REACH, co-creative interactions between humans and machines will be studied dynamically from different
perspectives (music sciences, social sciences, computer science) in order to highlight the combined conditions
under which these interactions occur, their temporal behavior, adaptive dynamics and control strategies, so as
to drastically enhance their creative potential.
We emphasize the fact that shared creativity is an emerging process, resulting from complex interactions andmultimodal, cross feed-back loops between natural and artificial actors, a process that cannot be reduced toan agent’s production considered in isolation, or to a single layer of technical skill. This conception neutralizes
the endless philosophical question of whether artificial entities can be qualified as “creative” and safely shifts
the REACH research interest towards the positive exploration of the best possible models and tools to allow
for co-creative cycles to bootstrap and settle.
We also believe that “computational creativity” will socially take root in a decisive way when it eventuallycreates the conditions of emergence of cyber-human co-creativity, and when it eventually clings to anaugmented physical reality allowing humans to experience a rewarding embodied relation. To this effect
REACH will work towards bridging the gap between abstract computational logics and mixed reality
experience anchored in creative musical instruments. In such configurations of physical interreality (a mixed
reality where the physical world is actively modified by human action), the human subjects participating toco-creative scenarios will be directly engaged in the digital, the physical and the social dimensions of theexperience. Such a seamless integration of cyber to physical to human will be the next frontier in Creative
Artificial Intelligence [Boden 1998, Briot & al 2019] and REACH aims at being a powerful instrument towards
it. So, REACH will implement interreality through Creative Instruments, music instruments that embed an
unheard level of digital intelligence and musicianship making them full partners of humans in co-creative
scenarios.

2AssayagPart B1

REACH

General background
In all sectors of human activity, we see coming new forms of blending between human creativity andautonomous information processing capabilities of physical objects that turn joint human machine interaction
into an augmented and mixed reality. In the cultural, artistic, educative or economical domains, co-creativity
between humans and machines will bring about the emergence of distributed information structures of a newkind, as in networked ambient intelligence or in artistic performances of mixed artificial and human agents.
These practices will disrupt known cultural orders and profoundly impact human development [Boden 1998][Assayag 2016]. With technologies allowing one to extract semantic features from physical (e.g. acoustic,
light) and human (e.g. physiological, gestural, vocal) signals [Dubos & al 2017] [Boyer & al 2017], combinedwith novel generative learning of symbolic representations [Briot & al 2019], we are beginning to unveil theincreasing complexity of cooperation, synergy or conflicts inherent to cyber-human networks. These newcooperative micro-societies need fresh conceptual and technological frames in order to be fully understood.
Our project aims at bringing a decisive contribution to solving this problem in the creative and artistic domains.
By understanding, modelling, and developing co-creativity between human and machines in the musicaldomain through improvised interactions REACH will allow musicians of any level of training to develop theirskills and expand their individual and social creativity. Indeed, improvisation is at the very heart of all humaninteractions, and music is a fertile ground for developing models and tools of creativity that can be generalized
to other activities, as in music the constraints are among the strongest to conduct cooperative behaviors thatmerge into highly efficient and meaningful courses of actions [Sawyer 2000]. REACH will study sharedmusicianship occurring at the intersection of the physical, human and digital spheres as an archetype of
distributed intelligence, and will produce powerful models and tools as vehicles to better understand andfoster individual and collective human creativity in a world where it becomes more and more intertwinedwith computation.
In addition to the exploration of computational creativity [Wiggins 2006, Colton and Wiggins 2012] in “soft”
human-computer configurations involving improvised musical interactions and displaying distributed cocreative behaviors, REACH assumes that cyber-human synergies will happen optimally when users are put ina situation where they experience full embodiment [Scurto & al 2017] in their relation to artificial agents,
through the physicality of familiar creative devices such as musical instruments. The musical market has
recently introduced cyber-physically augmented music instruments (Smart Instruments) such as the HyVibe
Guitar, that brings embedded computing, mechatronics and physical modelling straight under the fingers ofthe musician [Meurisse & al 2014]. Smart Instruments are a first step toward the blending of sophisticatedinformation processing power down to the core physics of the instruments in order to enhance their sounduniverse and their interactive power [Benacchio & al 2016]. However, at this stage, these devices do notdisplay musical creativity, in effect, they lack machine musicianship (the capacity to process and create music
information at a structural, semantic level), as well as digital intelligence in interaction strategies or generativeautonomy.
By bridging the gap between smart instruments with active acoustic control on the physical side, and digital
intelligence for co-creative interaction on the logical side, REACH will make possible the advent of Creative
Instruments: physical instruments loaded with active control and creative agents. A Creative Instrument will
be able to listen to the musician playing, learn music structures and style on-line and autonomously enter intoa creative dialog with the player. Such conditions will install a mixed reality framework, with cross-feedback
loops occurring through physical and perceptual channels between human and artificial agents, a novel
experience in artistic development.

Originality and impact
A major originality of this project is to constitute the first attempt to build a scientific and technological meshof knowledges and tools around computational co-creativity and musical improvisation through a highlyinterdisciplinary approach combining computational sciences of music, artificial intelligence, anthropology ofimprovised knowledges and practices, and mixed reality. It is also the first time that the notion of improvisationis brought as a general scheme of cyber-human communication targeting creative behaviors, and that methods
for augmenting human capacities and for enhancing computational creativity are brought together to model
symbiotic interaction. The REACH project will provide theoretical foundations, technologies, software suites
and experimental data and make them available in open-access to all the research community, and to othercommunities as well (e.g. artists, instrument makers, game developer, medias). REACH co-creative
instruments will bring about the long-awaited convergence between physicality, information processing andcreativity, in a manner that will be maneuverable and enjoyable by all, at all levels of skills and expertise. Such
3

AssayagPart B1

REACH

progress will be likely to disrupt artistic and social routines, as well as amateur and professional musicpractices, eventually impacting music industry in a game-changing way. As an illustration, in 2014, amateurand professional guitarists spent more than $4.25 billion on buying musical gear. However, it has become very
difficult to retain new buyers (90% of guitar buyers stop playing in the first year, according to Fender). By
adapting to individual skills, levels and musical tastes, Creative Instruments will increase the intensity andpleasure of creative use, contributing to human development and attaching users to artistic creativity in aconstantly renewed esthetic experience.
Cyber-human co-creativity is strongly felt when two features of improvisation linked to emergence [Canonne& Garnier 2011] and non-linear dynamics [Mouawad & Dubnov 2017] are present:ü emergence of cohesive concomitant behaviors that are not reducible to, nor explainable by the mere
individual processes of agents
ü apparition of non-linear regimes of event and structure formation, leading to rich musical co-evolution offorms.
Our assumption is that these surging phenomena result from cross learning processes between agents involving
complex feed-backs loops and reinforcement mechanisms, and our project will pioneer the way to create theoptimal conditions to accommodate these phenomena. A major consequence of co-creativity in that sense isthat it will then change the states and behavior of individual agents, and so participate in their evolution interms of knowledge and skills. To that extent, the vision of computational co-creativity foreseen by REACH
is totally novel with respect to state-of-the-art works, and specially with respect to recent heavy-weightresearch programs such as Google Magenta, Sony Flow Machines, MIT Media-Lab Opera of the future, orSpotify Creator Technology Lab (see e.g. [Roberts & al 2018], [Hadjeres & al 2017]), who are generally
centered on generative style learning and do not address directly, as we will, the co-action and emergenceproblem.
In REACH, human-machine co-creative strategies, based on the integration of artificial listening,interactive learning, structure discovery, mixed reality, and social science assessment, will provide a new
ground for understanding symbiotic interaction between the digital, the physical and the human world, withdeep repercussions on the unfolding of individual and collective creativity.

Methodologies
The fundamental working hypothesis behind REACH is that one has to operate in two symmetrical andconverging directions in order to simultaneously raise the capacity of human and cyber agents, so that theymeet at the point where synergistic effects may happen and make way for co-creative behavior. The two central
tasks (1) and (2) of the project will thus follow this scheme:
1. Raising Cyber Reach (RC): Augment the digital agents’ abilities with enhanced computational creativity,
autonomy and multimodal sensitivity to context, as well as with cyber-physical extensions in mixed reality
instruments, so they can enter into a convincing expert interplay with humans.
2. Raising Human Reach (RH): Augment the human abilities, by interfacing them into cross-modal,
embodied mixed reality so they can interact in a natural and creative way with artificial agents, and by
expanding their reach through novel collaborative strategies infused by social sciences field experiments.
In order to provide solid foundations and tools to sustain this double motion from Cyber to Human and from
Human to Cyber (1) and (2), we will surround it with two domains of action (3) and (4):
3. Deep Structure Discovery (DD): Machine Learning of complex multi-variate signals involving structure
discovery and knowledge extraction in musical signals
4. Probing Improvisation Practices (PP): Field research providing human (or combined cyber-human) data,experimental assessment and heuristic guides for model parametrization through an anthropology ofimprovisation practices, knowledges and processes.
RH and RC form the Epistemological Boost (EB) component of our research strategy: by symmetricallyaugmenting humans and artificial agents until they come together at their co-creative nexus by producing
emergent information, we will achieve an epistemological leap [Bachelard 1934] beyond the blocking ideathat artificial systems cannot be creative by themselves. This turn things around in stating that creativity is nota state anyway, but rather a dynamical effect of interaction in a complex system, likely to show that radical
novelty already identified as a characteristic of Emergence [Corning 2002]. By building on this epistemologicalboost, REACH will be able to model deep interactions that in turn will trigger co-creative behaviors.

4

AssayagPart B1

REACH

DD and PP form the Seeker Heads (SH): a cluster of exploratory researches in music structure discovery andimprovisation shaping, that will sustain and feed the EB component by providing general learning models,creation heuristics, behavioral examples, and social-science based protocols and data.
These two component EB and SH will be articulated by a Usage, Creation, and Outreach (UCO) package
acting as a project clock, or a heart-beat pulsating the timing of a series of iteration between modelling,
prototyping, experimenting, and creation / outreach. Going through fast research and development cycles
favoring interaction between all work packages, with regular milestones and deliverables, it will smooth-out
the interdisciplinary cooperation and prevent unforeseen problems that could potentially seize up theorganization of the project. UCO will clock at several nested periodicities (bimonthly, semester, year, biennial)
to schedule coordination meetings, delivery checking, scientific seminaries, local and far-reachinginternational workshops and dissemination events including tracks in major conference. Although REACH ishigh-risk high-gain, this heart-beat of animation, stimulation and coordination will provide a high level ofconfidence and security in smoothing the pace of research and production interaction, inside the PI’s teamand with external collaborators as well.

Inside this work program, we will use methods drawn mostly from (1) Interactive computational creativity, (2)mixed reality augmentation of music instruments (3) machine learning and musical information dynamics, and(4) social sciences and anthropology of improvised practices. Such a high level of inter-disciplinarity in the
REACH project is realistic since the PI has a unique background in heading for six years a world-renowned
laboratory encompassing all sciences of music and sound, where he has successfully organized wide-ranging,
multi-domain research strategies. The PI will nevertheless surround his team with a network of expert
researchers or economical actors, from which he will seek one-off expertise in well-defined perimeters, atvarious stages of the project, notably in the fields of Musical Information Dynamics and cognition, Social
Science / Anthropology, Augmented reality and Acoustics.
Interactive Computational creativity and mixed reality will be at the core of the Epistemological Boost (EB).
Improvised interaction between humans and digital agents [Blackwell 2012] is dependent on mastering
artificial creativity, which brings together several key research issues: knowledge, perception and intention
modeling, real-time interactive learning — featuring cross feed-backs between human and digital entities that
continuously modify the context (moving target)—, time structure formalization. Mixed reality will be
instantiated by Creative Instruments reassembling active acoustics and creative agents. Going beyond stateof-the-art cyber-physical systems (that create a continuity between the digital and the physical worlds),
REACH will promote cyber-human systems that create a continuity between digital and human creativity.
Machine Learning and Musical Information Dynamics of complex multi-variate signals such as polyphonicmusical signals, involving structure discovery and knowledge assessment, will be essential to understand andprocess (often in real-time) the variety of signals exchanged by humans and creative agents. These signals bear
implicit multi-dimensional and multi-scale semantic structures that are hard to tract computationally, but thatare nonetheless necessary for machine perception and learning to work properly. Representation learning ofthese structures, through artificial intelligence methods such as statistical learning and deep learning [Esling& al 2018] will be at the core of the Seeker Heads (SH). Musical Information Dynamics (MID) characterizesemantically important changes in information contents of audio signals [Abdallah & Plumbey 2009] [Dubnov
2010]. REACH will extend state-of-the-art MID towards longer term correlations and attention control, inorder to address the question of intention and decision over larger scale structures and bigger time spans. This
will bring a better understanding of joint cognition and creativity and will feed the design of creative agents.
It will also help us to assess that co-creativity is actually boosted by (EB), by checking information measures
5

AssayagPart B1

REACH

correlated to variations in surprise, interest, flow, affects, and analyzing their tendencies over different types
and scales of interactions.
Social Science, anthropological approaches will be used to observe, analyze, archive and understand
improvisational practices and knowledges between humans, as well as between humans and machines. Data
from these studies will help guide the conception of the perceptual, generative and interaction models in EB
and DD. This work at the interface of technology, artistic practices and urban anthropology, will proceed onlive performance involving different modes of human-machine interaction, in diverse cultural background. It
will use the methods of the field survey in the usual sense of ethnomusicology: performance data collecting,audio-visual and gestural capture of live context of musical sessions, interviews with participants (musicians,
audience), sessions of re-listening, analysis of the musical contents archived by the software prototypes. A
series of questions will be examined: How do the musicians react to the computer actions? Does it influence
the way human musicians play themselves? Do they perceive computer productions as intentional, on-purpose?
Do they perceive computer interaction as realistic in the sense that humans could produce it, or rather as unhuman, thus potentially bending towards totally unheard artistic forms and aesthetics? This will bringunderstanding of joint cognition and action from the human science angle and will complement and correlate
the “objective” MID assessment of co-creativity emergence, by “subjective” survey of human behavior anddiscourses in creative situations.

Feasibility
Host Institution (HI) IRCAM, founded by famous composer and conductor Pierre Boulez, is the largest
institution in the world gathering scientific research, technology engineering, contemporary music production,
concert season and teaching. It offers absolutely unique conditions for REACH researches that seat at thecrossing of several scientific, aesthetic, social and technological domains. The hosting Lab STMS (Sciences
and technologies of music and sound, a joint operation by IRCAM, CNRS and Sorbonne University) inside
IRCAM is world leader in Sound and Music Computing. The hosting team in STMS, called Music
Representation, is headed by the PI and internationally renowned in Musical Informatics and Creative
Interaction.
The REACH team itself will be formed by the PI (40% of FTE time) plus three postdocs (PoD), three PhD
students and a research assistant. The couples (PoD1, PhD1) will focus on the EB cluster, (PoD2, PhD2) on
the PP task, and (PoD3, PhD3) on the DD task, with several interactions and crossings. The hosting lab will
provide a great amount of punctual expertise as needed in the different tasks of the project Pr. Philippe Eslingis an expert in Creative Artificial Intelligence and the founder of the ACIDS research group (Artificial Creative
Intelligence and Data Science). The S3AM team (acoustics, mechatronics, signal processing, and physicalmodelling, head: T. Hélie), ISMM team (embodiment, human gesture, head: F. Bevilacqua), and PDS(cognition, perception, computational neurosciences, head: P. Susini) are recognized leaders in their domain
and will provide invaluable punctual expertise in these fields. The IRCAM Music Production and Teaching
Department, used to manage international seasons of artistic productions, will be decisive in supporting the PI
for experimental sessions and artistic outreach.
Regular international collaborators of the PI in Europe, USA and Japan, will provide punctual expertise orservice delivery where needed : the award winning Hyvibe high tech-sound company, on prototyping andexperimenting the Creative Instruments; UCSD Lab CREL (Center for Research in Entertainment andLearning, (Dir. Shlomo Dubnov, a recognized leader in Musical Information Dynamics) on machine learning,
artificial listening and edutainment applications ; EPFL Meta Media Center / Montreux Jazz Heritage Lab(Resp. Alain Dufaux) will give access to huge amounts of annotated live musical data through the Heritage
Montreux Jazz Festival Archive, a registered UNESCO Immaterial Memory of the World Heritage; EHESSCAMS Lab (Ecole des Hautes Etudes en Sciences Sociales, Centre d’Analyse et de Mathématique Sociales,
PI: Marc Chemillier) will provide social science / anthropological method and expertise for the PP task. The
PI also maintain continuous collaboration and PhD students exchange with several labs at Tokyo University(Pr. Harada Cyber Physical Systems Lab, Pr. Hirose Cyber Laboratory) and Tokyo Metropolitan University(Pr. Ikei Virtual Reality and Ultra Reality Lab) who will provide expertise on cyber-physical systems, mixed
reality and deep learning theory and practice.

6

AssayagPart B1

REACH

References
Abdallah S.A. and M. Plumbley, D., Information dynamics: patterns of expectation and surprise in theperception of music, Connection Science, Vol. 21, No. 2. (2009), pp. 89-117.
Allombert, A., Desainte-Catherine, M., Assayag, G., « Towards an hybrid temporal paradigm for musicalcomposition and performance », Computer Music Journal, 2012, vol. 4, n° 36
Allombert, A., Desainte-Catherine, M., and Assayag, G. Iscore: a system for writing interaction. Proc.3rd
international Conference on Digital interactive Media in Entertainment and Arts, DIMEA '08, vol. 349,
360-367. ACM, New York, 2008
Allombert A., Desainte-Catherine M., Larralde J. and Assayag G., «A system of Interactive Scores based on
qualitative and quantitative temporal constraints», Proc. of the 4th International Conference on Digital
Arts (Artech 2008), Porto, 2008.
Allwood, J., Linguistic communication as action and cooperation. Gothenburg monographs in linguistics,
1976, vol. 2, p. 637-663.
Agon, C., Bresson, J., Assayag, G., OpenMusic: Design and Implementation Aspects of a Visual Programming
Language », 1st European Lisp Symposium - ELS'08, Bordeaux, 2008
Allombert, A., Assayag, G., Desainte-Catherine, M., « A system of Interactive Scores based on Petri Nets »,SMC'07 - 4th Sound and Music Computing Conference, Lefkada, pp. 158-165, 2007
Andreatta M., Musique algorithmique, In N. Donin et L. Feneyrou (eds), Théorie de la composition musicale
au XXe siècle, Symétrie, Paris, 2013
Arom, S., African Polyphony and Polyrhythm: Musical Structure and Methodology, Cambridge University
Press, 1991 - 668 pages.
Argamon, S., Burns ,K., and Dubnov, S., (eds), “The Structure of Style: Algorithmic Approaches toUnderstanding Manner and Meaning”, Springer SBM, 2010
Assayag, G., Improvising in Creative Symbolic Interaction. In Smith; Chew; Assayag (Eds), Mathematical
Conversations: Mathematics and Computation in Music Performance and Composition, World Scientific;
Imperial College Press, pp.61 - 74, 2016
Assayag,, G. Creative Symbolic Interaction, 40th Intl. Comp. Mus. Conf. and 11th Sound and Music Comp.
Conf. (ICMC / SMC joint conf.), Athenes, Greece, pp 1-6, 2014.
Assayag, G., Truchet, C. (Eds) « Constraint Programming in Music », Constraint Programming in Music, ISTE
Ltd and John Wiley & Sons Inc, 256 p., 2011
Assayag, G., Bloch, G., Cont, A., Dubnov, S., Interaction with Machine Improvisation, in S.Argamon, S.
Dubnov and K.Burns (Eds) The Structure of Style: Algorithmic Approaches to Understanding Mannerand Meaning, Springer, pp.219-245, 2010
Assayag, G., Gerzso, A., « New Computational Paradigms for Computer Music », New Computational
Paradigms for Computer Music, ed. Gérard Assayag et Andrew Gerzso. (Ircam/Delatour), 2009
Assayag, G., Bloch, G., Navigating the Oracle: a Heuristic Approach, International Computer Music
Conference '07, Copenhagen, 2007, pp. 405-412
Assayag, G., Bloch, G., Chemillier, M., OMax-Ofon, Sound and Music Computing SMC’06, Marseille, 2006
Assayag, G., Bloch, G., Chemillier, M., Improvisation et réinjection stylistiques. Le feed-back dans la création
musicale contemporaine - Rencontres musicales pluri-disciplinaires, Mar 2006, Lyon, France.
Assayag, G., Bloch, G., Chemillier, M., Cont, A., Dubnov, S., Omax Brothers : a Dynamic Topology of Agentsfor Improvisation Learning, Audio and Music Computing for Multimedia, ACM Multimedia ’06, Santa
Barbara, 2006
Assayag, G., Dubnov, S., « Improvisation Planning and Jam Session Design using concepts of Sequence
Variation and Flow Experience », Sound and Music Computing 2005, Salerno, 2005
Assayag, G., Dubnov, S., Using Factor Oracles for machine Improvisation, Soft Computing, Springer Verlag,
2004, 8 (9), pp.604-610, 2004.
Assayag, G., Dubnov, S., Delerue, O., Guessing the Composer's Mind : applying Universal Prediction toMusical Style, ICMC: International Computer Music Conference, Beijing, 1999
Assayag, G., Agon, C., Fineberg, J., Hanappe, P., An Object Oriented Visual Environment for MusicalComposition, ICMC: International Computer Music Conference, Thessaloniki, 1997, pp. 364-367
Assayag G., Agon, C., OpenMusic Architecture. International Computer Music Conference ICMC’96, Hong
Kong 1996.
7

AssayagPart B1REACH

Assayag, G., Visual Programming in Music, International Computer Music Conference ICMC’95, Banff,
1995, pp. 73-76
Arias, T., Desainte-Catherine M., and Dubnov, S., Automatic Construction of Interactive Machine
Improvisation Scenarios from Audio Recordings, 4th Intl Workshop on Musical Metacreation,
International Conference on Computational Creativity, Paris, 2016
Auslander, Ph. Liveness: Performance in a mediatized culture. New York: Routledge, 1999.
Ayad, L., Chemillier, M., Pissis, S., , Creation Improvisation on Chord Progressions using Suffix Trees, Journalof Mathematics and Music, vol 12 (3), 2018.
Bachelard, G., La formation de l’esprit scientifique, Paris, Alcan, 1934 (reprint. Paris PUF coll. « Quadrige »,2013)
Bamberger, J., Discovering the musical mind: A view of creativity as learning. New York, NY: Oxford
University Press, 2013.
Ban, Y.,, T. Narumi, T. Fujii, S. Sakurai, J. Imura, T. Tanikawa and M. Hirose: Augmented Endurance:
Controlling Fatigue while Handling Objects by Affecting Weight Perception using Augmented Reality,
CHI2013, pp.69-78. 2013.
Boden, M., Computers models of creativity. AI Magazine 30(3): 23–34. 2009
Boden M.A., Creativity and Artificial Intelligence. Artificial Intelligence, 103:347-356, 1998.
Boden, M.A., Dimensions of Creativity, M. A. Boden (ed.), The. MIT Press, Cambridge, Mass., 1996
Boden, M.A., The Creative Mind: Myths and Mechanisms. New York: Basic Books. 1991
Bailey, Derek, L'improvisation. Sa nature et sa pratique dans la musique, trad. Isabelle Leymarie, Paris, Outre
Mesure, 1999.
Berliner, Paul. Thinking in Jazz: The Infinite Art of Improvisation. Chicago: University of Chicago Press,
1994.
Benacchio, S., Mamou-Mani, A., Chomette, B., Finel, V. Active control and sound synthesis—two different
ways to investigate the influence of the modal parameters of a guitar on its sound. Journal of the
Acoustical Society of America, 139 (3), pp.1411, 2016
Benacchio, S., Mamou-Mani, A., Chomette, B., Causse, R. Active control applied to string instruments.
Société Française d'Acoustique. Acoustics 2012, Apr 2012, Nantes, France. 2012.
Bengio, Y., Boulanger-Lewandowski, N., & Pascanu, R. (2012). Advances in optimizing recurrent networks.
2013 IEEE International Conference on Acoustics, Speech and Signal Processing, (2012): 8624-8628.
Bimbot, F., E. Deruty, G. Sargent, and E. Vincent. System & Contrast: a polymorphous model of the innerorganization of structural segments within music pieces. Music Perception 33(5):631–661. 2016
Bigo, L., M. Giraud, R. Groult, N. Guiomard-Kagan, and F. Levé. “Sketching sonata form structure inselected classical string quartets.” In Proceedings of the International Society for Music Information
Retrieval Conference. pp. 752–759. 2017.
Birkhoff, G.D., Aesthetic Measure, Harvard Univ. Press, 1933.
Blackwell, T., Oliver Bown & Michael Young. Live Algorithms: Towards Autonomous Computer Improvisers.
In Jon McCormack & Mark d’Inverno, editeurs, Computers and Creativity, pages 147–174. Springer
Berlin Heidelberg, 2012.
Bloch, G., Dubnov, S., Assayag, G., « Introducing Video Features and Spectral Descriptors in the OMax
Improvisation System », International Computer Music Conference ICMC’08, Belfast, 2008
Butler, M. Playing with Something That Runs. Technology, Improvisation, and Composition in DJ and Laptop
Performance, Oxford University Press, 2014.
Bonnasse-Gahot, L., An update on the SOMax project, IRCAM Report, Soma v0.1.3 (2014)
Boyer, E., Arthur Portron, Frédéric Bevilacqua, Jean Lorenceau. Continuous Auditory Feedback of Eye
Movements: An Exploratory Study toward Improving Oculomotor Control. Frontiers in Neuroscience,
2017, 11
Briot, J.-P. , Hadjeres G., and Pachet, F.D., Deep Learning Techniques for Music Generation, Computational
Synthesis and Creative Systems, Springer, 2019
Bresson, J., Guédy, F., Assayag, G., Musique Lab 2: From computer-aided composition to music education,
Journal of Music, Technology and Education, Intellect, 2013, 5 (3), pp.273-291.
Bresson J., Agon C., Assayag G., Visual Lisp/CLOS Programming in OpenMusic. Higher-Order and Symbolic
Computation. Mars 2009, vol. 22, n° 1, p. 81-111
Canonne, C., Garnier, N., "A Model for Collective Free Improvisation", Mathematics and Computation in
8AssayagPart B1

REACH

Music. Third International Conference MCM 2011, IRCAM, Paris, France, June 15-17, 2011.
Proceedings, Springer, 2011.
Canonne, C., Garnier, N., Cognition and Segmentation In Collective Free Improvisation: An Exploratory
Study. International Conference on Music Perception and Cognition, 2012, Thessaloniki, Greece.
Carpentier, G., Assayag, G., Saint-James, E. « Solving the Musical Orchestration Problem using
Multiobjective Constrained Optimization with a Genetic Local Search Approach », Journal of Heuristics,
2009.
Carpentier, G., Tardieu, D., Assayag, G., Rodet, X., Saint-James, E., (2007) « An Evolutionary Approach toComputer-Aided Orchestration », EvoMUSART, vol. LNCS 4448, Valence, 2007, pp. 488-497
Carsault, T. Nika, J. & Esling, P. Using musical relationships between chord labels in Automatic Chord
Extraction tasks. 19th ISMIR Conference (ISMIR2018), Paris, France, 2018
Clayton, Martin. What is entrainment? Definition and applications in musical research. Empir. Musicol. Rev.
7, (2012) 49–56.
Chemillier, M. De la simulation dans l'approche anthropologique des savoirs relevant de l'oralité : le cas de lamusique traité avec le logiciel Djazz et le cas de la divination, Transposition, Hors-série n°1, Musique,histoire, société, 2018
Chemillier, M. Nika Jérome, « Étrangement musical » : les jugements de gout de Bernard Lubat à propos dulogiciel d'improvisation ImproteK, Cahiers d'ethnomusicologie, n° 28, 2016, pp. 61-80.
Chemillier, M. Jazz et... musiques électroniques, P. Carles, A. Pierrepont (éds.), Polyfree. La jazzosphère (etailleurs) : une histoire récente (1970-2015), Paris, Outre Mesure, ch. 3, pp. 43-54, 2016.
Chemillier, M., Ruse et combinatoire tsigane. De la modélisation informatique dans les répertoiresmusicaux
traditionnels (à propos du livre de Victor A. Stoichita : Fabricants d'émotion. Musique et malice dans unvillage tsigane de Roumanie), L'Homme, n° 211, 2014, pp. 117-128.
Chemillier, M., Fieldwork in Ethnomathematics, Nick Thieberger (ed.), The Oxford Handbook of Linguistic
Fieldwork, Oxford University Press, 2011, chapter 14, pp. 317-344.
Chemillier, M. L'improvisation musicale et l'ordinateur. Transcrire la musique à l'ère de l'image animée,Terrain, n° 53 « Voir la musique », 2009, pp. 67-83.
Chemillier, M., Toward a formal study of jazz chord sequences generated by Steedman’s grammar. Soft
Computing, Vol. 8, No. 9, pp. 617–622, 2004.
Chemla-Romeu-Santos, A., Ntalampiras, S., Esling, P., Haus G., and Assayag, G., Cross-Modal Variational
Inference for Bijective Signal-Symbol Translation, 22nd International Conference on Digital Audio
Effects DAFx 2019
Chew, E.. From sound to structure: Synchronising prosodic and structural information to reveal the thinking
behind performance decisions. In C. Mackie (ed.): New Thoughts on Piano Performance, selected papers
from the 2013 and 2015 London International Piano Symposium. Kindle Amazon, 2017
Chew, E. Mathematical and Computational Modeling of Tonality: Theory and Applications. International
Series in Operations Research and Management Science, New York, NY: Springer, 2014
Cholakis, Ernest, Sound Analysis of Swing in Jazz Drummers: An Analysis of Swing Characteristics of 16
well known Jazz Drummers, 1995, http://www.numericalsound.com/sound-analysis.html
Collins, D. A synthesis process model of creative thinking in music composition. Psychology of Music, 33(2):193-216, 2005.
Cook, N., Beyond the Score: Music as Performance. Oxford, UK: Oxford University Press, 2013
Conklin, D., Multiple viewpoint systems for music classification. Journal of New Music Research, Volume 42,
Number 1, pp. 19-26, March 2013
Conklin, D. Music Generation from Statistical Models , Proceedings of the AISB 2003 Symposium on
Artificial Intelligence and Creativity in the Arts and Sciences, Aberystwyth, Wales, 30– 35, 2003.
Crestel, L. Esling, P. Ghisi, D. Meier, R. Generating orchestral music by conditioning SampleRNN, Timbre
2018: Timbre is a many-splendored thing. Montreal, 2018.
Crestel, L., Esling. P., Live Orchestral Piano, a system for real-time orchestral music generation. 14th Soundand Music Computing Conference 2017, Jul 2017, Espoo, Finland. pp.434, 2017
Crochemore, M., Ilie L., Seid-Hilmi, E., The Structure of Factor Oracles, Int. J. Found. Comput. Sci. 18(4),781–797 (2007).
Colton, S. and Wiggins, G. A.. Computational creativity: The final frontier? In Proceedings of 20th European
Conference on Artificial Intelligence (ECAI), pages 21-26, 20129

AssayagPart B1

REACH

Cont, A., Dubnov, S., Assayag, G., On the Information Geometry of Audio Streams with Applications to
Similarity Computing, IEEE Trans. on Audio, Speech and Language Processing, 19 (4), pp.837-846, 2011
Cont, A., Dubnov, S., Assayag, G., GUIDAGE: A Fast Audio Query Guided Assemblage, International
Computer Music Conference, ICMC’07, Copenhagen, 2007
Cont, A., Dubnov, S., Assayag, G., Anticipatory Model of Musical Style Imitation using Collaborative andCompetitive Reinforcement Learning , in Butz, M., Sigaud O., and Baldassarre G. (eds.), Anticipatory
Behavior in Adaptive Learning Systems, Lecture Notes in Computer Science, Springer Verlag, Berlin,
vol. 4520, pp. 285-306, 2007
Cont, A., Dubnov S., Assayag, G., A Framework for Anticipatory Machine Improvisation and Style Imitation”,
Adaptive Behavior in Anticipatory Learning Systems / Simulation of Adaptive Behavior (SAB 2006)
Rome, Italy, 2006.
Corning, P. A., The Re-Emergence of "Emergence": A Venerable Concept in Search of a Theory, Complexity,
7 (6): 18–30, 2002
Daikoku, T., Musical Creativity and Depth of Implicit Knowledge: Spectral and Temporal Individualities in
Improvisation. Frontiers in Computational Neuroscience vol.12. 2018.
Déguernel, K., Vincent, E., Assayag, G., Learning of Hierarchical Temporal Structures for Guided
Improvisation. Computer Music Journal, accepted 2019 .
Déguernel, K., Vincent, E., Assayag, G., Probabilistic Factor Oracles for Multidimensional Machine
Improvisation. Computer Music Journal, 42:2, article 4, Summer 2018
Deguernel, K., Nika, J., Vincent, E., Assayag, G. Generating Equivalent Chord Progressions to Enrich Guided
Improvisation : Application to Rhythm Changes. SMC 2017 - 14th Sound and Music Computing
Conference, Espoo, Finland. pp.8, 2017
Deguernel, K., Nika, J., Vincent, E., Assayag, G., Using Multidimensional Sequences For Improvisation In
The OMax Paradigm. 13th Sound and Music Computing Conference, Hamburg, Germany, 2016.
Doffman, Mark, Groove! Its Production, Perception, and Meaning in Jazz, M.A. Thesis, U. of Sheffield, 2005.
Doras, G., Esling, P., & Peeters, G. On the use of u-net for dominant melody estimation in polyphonic music.
IEEE International Workshop on Multilayer Music Representation, Milano, Italy, pp. 66-70 (2019)
Donze, A., S. Libkind, S.A. Seshia, D. Wessel, Control improvisation with application to music. Technicalreport No. UCB/EECS-2013-183. EECS Department, University of California, Berkeley, 2013.
Dubnov, S., Characterizing Time Series Variability and Predictability from Information Geometry Dynamic,
vol. 8085 of the Lecture Notes in Computer Science series, pp. 658-668, Springer, 2013
Dubnov,S., Musical Information Dynamics as Models of Auditory Anticipation, in Machine Audition:
Principles, Algorithms and Systems, ed. W. Weng, IGI Global publication, 2010
Dubnov, S., Unified View of Prediction and Repetition Structure in Audio Signals with Application to Interest
Point Detection, IEEE Trans. on Audio, Speech and Language Proc., Vol. 16, No. 2, pp. 327-337, 2008
Dubnov, S., Assayag, G., Music Design with Audio Oracle using Information Rate. First International
Workshop On Musical Metacreation (Mume 2012), Oct 2012, Palo Alto, United States. pp.1-1, 2012.
Dubnov, S., Assayag, G., Cont, A., Audio Oracle Analysis of Musical Information Rate, Proceedings of IEEE
Semantic Computing Conference (ICSC2011), United States. pp.567—571, 2011
Dubnov, S., Assayag, G., Cont, A., Audio Oracle: A New Algorithm for Fast Learning of Audio Structures,
International Computer Music Conference, Copenhaguen, 2007
Dubnov, S., Assayag, G., Lartillot, O., Bejerano G., Using Machine-Learning Methods for Musical Style
Modeling, IEEE Computer, vol. 10, n° 38, Octobre, 2003
Dubnov, S., Assayag, G., Universal Prediction Applied to Stylistic Music Generation, in Assayag, G.,
Feichtinger, H.G., Rodrigues, J.F. (eds.), Mathematics and Music, Springer-Verlag, Berlin, pp. 147-158,
2002
Dubnov, S., Burns, K.,Kiyoki, Y., Cross-Cultural Multimedia Computing - Semantic and Aesthetic Modeling.
Springer Briefs in Computer Science, Springer 2016
Dubnov, S., Cont, A., Assayag, G., Audio Oracle: A New Algorithm for Fast Learning of Audio Structures »,International Computer Music Conference, Copenhaguen, 2007
Dubnov, S., McAdams, S., and Reynolds, R., Structural and Affective Aspects of Music from Statistical Audio
Signal Analysis, Journal of the American Society for Information Science and Technology, 57(11):1526–
1536, 2006
Dubnov S., and Surges, G., Delegating Creativity: Use of Musical Algorithms in Machine Listening and
10AssayagPart B1

REACH

Composition, in Digital da Vinci, pp. 127-157, Springer, 2013
Dubos, A., Frédéric Bevilacqua, Joseph Larralde, Joël Chevrier, Jean-François Jego. Designing Gestures forInteractive Systems: Towards Multicultural Perspectives. 16th IFIP Conference on Human-Computer
Interaction (INTERACT), Sep 2017, Bombay, India. pp.524-526.
Echeveste, J., Cont, A., Giavitto, J.-L., Jaquemard, F. Operational semantics of a domain specific language forreal time musician–computer interaction. J. of Discrete Event Dynamic Systems, 2013, vol. 23, n°4, pp
343-383.
Elgammal, A., Liu, B., Elhoseiny, M., Mazzone, M.: CAN: Creative adversarial networks generating “art” bylearning about styles and deviating from style norms (2017). ArXiv:1706.07068v1
Esling, P., Chemla-Romeu-Santos, A., & Bitton, A. “Generative timbre spaces: regularizing variational autoencoders with perceptual metrics”. Proceedings of International DaFX conference, pp.23-27, Portugal(2018)
Esling, P. Chemla, A. Bitton, A. “Bridging audio analysis, perception and synthesis with perceptuallyregularized variational timbre spaces”, Proc. 19th International Society for Music Information Retrieval,vol. 1, Paris, France. AR: 23,2% Best paper award. (2018)
Fujie, S., Yagi, D., Matsusaka, Y., et al. Spoken dialogue system using prosody as para-linguistic information.
In : Speech Prosody 2004, International Conference. 2004.
Fitch WT. Rhythmic cognition in humans and animals: distinguishing meter and pulse perception. Front Syst
Neurosci. 2013;7:68.
François, A. R. J., Schankler, I., and Chew E., Mimi4x: An interactive audio-visual installation for high-levelstructural improvisation. International Journal of Arts and Technology, 6(2): 138–151, 2013.
François, A.R.J., E. Chew, D. Thurmond (2008). Visual Feedback in Performer-machine Interaction forMusical Improvisation. In Proc NIME, 277-280.
Gibson, W., The group ethic in the improvising jazz ensemble: A symbolic interactionist analysis of music,
identity, and social context. Studies in Symbolic Interaction. 2010.
Gilhooly, K. J., Thinking: Directed, Undirected and Creative. London: Academic Press, 1996.
Giraud, M., R. Groult, E. Leguy, and F. Levé. 2015. “Computational fugue analysis.” Computer Music Journal
39(2):77–96.
Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozairy, S., Courville, A., Bengio, Y.,
Generative adversarial nets (2014). ArXiv:1406.2661v1
Hadjeres, G., Pachet, F., Nielsen, F.: DeepBach: a steerable model for Bach chorales generation (2017).
ArXiv:1612.01010v2
Herremans, D., Chuan, C.H., Chew, E.: A functional taxonomy of music generation systems. ACM Computing
Surveys (CSUR) 50(5) (2017)
Hinton, G.E.: Boltzmann machine. Scholarpedia 2(5), 1668 (2007)
Kato, H., Ushiku, Y., Harada. T. Neural 3D Mesh Renderer. The 31th IEEE Computer Society Conference on
Computer Vision and Pattern Recognition (CVPR), 2018.
Ikei, Y, Hirota, K., Amemiya T., Kitazaki, M., Five Senses Theatre: A multisensory display for the bodily Ultra
Reality, Emotional Engineering Vol.4, (Ed.) Shuichi Fukuda, Springer, 2016
Ishi, C. T., Ishiguro, H., et Hagita, N.. Analysis of relationship between head motion events and speech indialogue conversations. Speech Communication, 2014, vol. 57, p. 233-243.
Kanehira, A., Takemoto, K., Inayoshi, S., Harada, T., Multimodal Explanations by Predicting
Counterfactuality in Videos. The 32nd IEEE Computer Society Conference on Computer Vision and
Pattern Recognition (CVPR), 2019.
Keil, Charles. The Theory of Participatory Discrepancies: a progress report. Ethnomusicology, 39(1), (1995)
1-19.
Kingma, D.P., Welling, M., An Introduction to Variational Autoencoders, arXiv:1906.02691, 2019
Kingma, D.P., Welling, M.: Auto-encoding variational Bayes (2014). ArXiv:1312.6114v10
Knapp, M., Hall, J., Nonverbal Communication in human interaction, Cengage Learning, 2013.
Kopp, S., Social resonance and embodied coordination in face-to-face conversation with artificial
interlocutors. Speech Communication, 2010, vol. 52, no 6, p. 587-597.
Large, Edward W., and John F. Kolen. 1994. Resonance and the perception of musical meter. Connection
Science 6(2–3):177–208.
Large, Edward W., and Caroline Palmer. 2002. Perceiving temporal regularity in music. Cognitive Science11

AssayagPart B1

REACH

26:1–37
Lartillot, O., Dubnov, S., Assayag, G., Bejerano, G., Automatic Modeling of Musical Style, International
Computer Music Conference, La Havane, 2001, pp. 447-454
Lévy, B., Bloch, G., Assayag, G., « OMaxist Dialectics : capturing, Visualizing and Expanding Improvisations», NIME12, Ann Arbor, 2012, pp. 137-140
Lissek, H., Rivet, E., Laurence, T., Fleury, R., Toward Wideband Steerable Acoustic Metasurfaces with Arraysof Active Electroacoustic Resonators; Journal of Applied Physics. 2018.
McPherson, A.P. Kim, Y., Augmenting the Acoustic Piano with Electromagnetic String Actuation andContinuous Key Position Sensing. Proc. Intl. Conference on New Interfaces for Musical Expression
NIME’10, 2010
Maniatakos, F., Assayag, G., Bevilacqua, F., Agon, C., On architecture and formalisms for computer assisted
improvisation , Proc. Sound and Music Computing conference, SMC’10, Barcelona, 2010
Mehri, S., Kumar, K., Gulrajani, I., Kumar, R., Jain, S., Sotelo, J., Courville, A. & Bengio, Y. (2016).
SampleRNN: An Unconditional End-to-End Neural Audio Generation Model (cite arxiv:1612.07837)
Meurisse, T., Mamou-Mani, A., Causse, R., Chomette, B., Sharp, D. Simulations of modal active control
applied to the self-sustained oscillations of the clarinet. Acta Acustica, Hirzel Verlag, 2014, 100(6),pp.1149-1161.
Meyer, L.B., Emotion and Meaning in Music. Chicago: Chicago University Press, 1956.
Moles, A., Théorie de l’information et perception esthétique, Flammarion, Paris, 1958
Moles, A., Art et ordinateur, Pris Casterman, 1971.
Monson, Ingrid. Saying something: jazz improvisation and interaction.University of Chicago Press, 1996.
Moreira, J., Roy, P., Pachet, F. VirtualBand: Interacting with Stylistically Consistent Agents.ISMIR, pages 341346, Curitiba (Brazil), 2013
Mouawad, P., Dubnov, S., On Modeling Affect in Audio with Non-Linear Symbolic Dynamics, Advances inScience, Technology and Engineering Systems Journal, Vol. 2, No. 3, 1727-1740 2017
Mouawad, P., Dubnov, S., On Symbolic Dynamics and Recurrence Quantification Analysis for Affect
Recognition invoice and Music, Perspectives in Nonlinear Dynamics 2016, Berlin
Nika, J., Chemillier, M., ImproteK, integrating harmonic controls into improvisation in the filiation of OMax,
Proceedings of the International Computer Music Conference, pp. 180–187, 2012.
Nika, J., Chemillier, M., Assayag, G., ImproteK: introducing scenarios into human-computer music
improvisation, ACM Computers in Entertainment, 14 : 2, article 4, 2017.
Nika, J., Deguernel, K., Chemla--Romeu-Santos, A., Vincent, E., Assayag, G., DYCI2 agents: merging the"free", "reactive", and "scenario-based" music generation paradigms, Proc. ICMC, Shangai, China, 2017
Novielli, N., De Rosis, F., and Mazzotta, I., User attitude towards an embodied conversational agent: Effectsof the interaction mode. Journal of Pragmatics, 2010, vol. 42, no 9, p. 2385- 2397.
Ogawa, N., Y. Ban, S. Sakurai, T. Narumi, T. Tanikawa and M. Hirose: Metamorphosis Hand: DynamicallyTransforming Hands, ACM Augmented Human 2016, 2016.
Pachet, F. and Roy, P. Markov constraints: steerable generation of Markov sequences. Constraints, 16(2):148172, March 2011
Pachet, F., Musical virtuosity and creativity. Computers & Creativity, 2012.
Patel, Aniruddh D., John R. Iversen, Micah R. Bregman, Irena Schulz, and Charles Schulz. 2008. Investigating
the human-specificity of synchronization to music. Proc.10th Intl. Conf. on Music Perception andCognition 1–5, 2008.
Pinch, T., Trocco, F. Analog days: The invention and impact of the Moog synthesizer. Cambridge, MA:
Harvard University Press, 2002.
Potter, K., Wiggins, G., and Pearce, M., Towards greater objectivity in music theory: Information-dynamic
analysis of minimalist music. In Musicae Scientiae 11(2), 295-322, 2007
Pressing, J., Cognitive processes in improvisation. Advances in Psychology, Vol. 19, pp. 345–363, 1984.
Rabinovich, M.I, and Varona, P., (2016), Consciousness and joint Human-Robot Creativity; Chaotic Chunking,
Frontiers Neuroscience 2016
Rabinovich, M., Trisan I., and Dubnov, S., Nonlinear dynamics of human creativity, IEEE International
Conference on Systems, Man and Cybernetics, 2014
Roberts, A., Engel, J., Raffel, C., Hawthorne, C., Eck, D.: A hierarchical latent vector model for learning long12

AssayagPart B1

REACH

term structure in music. In: Proceedings of the 35th International Conference on Machine Learning (ICML2018). ACM, Montreal, PQ, Canada (2018)
Rowe, R., Machine Musicianship. MIT Press, Cambridge, MA, USA, 2004.
Rueda, C., Assayag, G., Dubnov, S., A Concurrent Constraints Factor Oracle Model for Music Improvisation», XXXII Conferencia Latinoamericana de Informática CLEI 2006, Santiago, 2006
Sanlaville, K., Assayag, G., Bevilacqua F., Pelachaud, C., Emergence of synchrony in an Adaptive Interaction
Model, Intelligent Virtual Agents 2015 Doctoral Consortium, 2015, Delft,
Sawyer, R.K., Improvisational Cultures: Collaborative Emergence and Creativity in Improvisation, Mind,
Culture, and Activity, 7:3, 2000
Schankler, I., Chew, E., and François A. R. J., Improvising with digital auto-scaffolding: How Mimi changes
and enhances the creative process. In Digital Da Vinci, ed. Newton Lee, pp. 99–125. Berlin: Springer
Verlag. 2014.
Scurto, H., Frédéric Bevilacqua. VIMOs: Enabling Expressive Mediation and Generation of Embodied
Musical Interactions. 4th International Conference on Movement Computing, Jun 2017, London.
Smith, J., Chew, E., Assayag G., (Eds), Mathemusical Conversations : Mathematics and Computation in Music
Performance and Composition, Feb 2015, Singapore. World Scientific; Imperial College Press, 2016.
Smith, J. B. L., Schankler, I., Chew, E., Listening as a Creative Act: Meaningful Differences in Structural
AnnotationsofImprovised
Performances.
Music
TheoryOnline,20(3).
2014.
www.mtosmt.org/issues/mto.14.20.3/mto.14.20.3.smith_schankler_chew.php
Steels, L. Evolving grounded communication for robots. Trends in cognitive sciences, 2003, vol. 7, no 7, p.
308-312.
Smith, J. B. L., Chuan, C., Chew, E., Audio Properties of Perceived Boundaries in Music. IEEE Transactions
on Multimedia Special Issue on Music Data Mining, 16(5): 1219-1228, 2014.
Surges, G. and Dubnov, S. Feature Selection and Composition using PyOracle. Workshop on Musical
Metacreation, Ninth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment.
Northeastern University, Boston, MA. October 14-15, 2013.
Thórisson, K. R. Natural turn-taking needs no manual: Computational theory and model, from perception toaction. In : Multimodality in language and speech systems. Springer Netherlands, 2002. p. 173- 207.
Tokushige, H., T. Narumi, S. Ono, Y. Fuwamoto, T. Tanikawa, and M. Hirose: Trust lengthens decision timeon unexpected recommendations in human-agent interaction, HAI2017, 2017.
Tokozume, Y., Ushiku, Y., Harada, T., Learning from Between-class Examples for Deep Sound Recognition.
The 6th International Conference on Learning Representations (ICLR 2018), 2018
Tokozume, Y., Harada, T., Learning environmental sounds with end-to-end convolutional neural network. The
42nd IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2017), 2017
Toro M., Rueda, C., Agon, C., Assayag, G., GELISP: A Framework to represent musical constraint satisfaction
problems and search strategies , Journal of Theoretical and Applied Information Technology, JATIT, 2016,
86 (2), pp.327-331.
Toro M., Rueda, C., Agon, C., Assayag, G., NTCCRT: A Concurrent Constraint Framework for Soft RealTime Music Interaction, Journal of Theoretical and Applied Information Technology, JATIT, 2015, 82 (1),
pp.184-193.
van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A.,
Kavukcuoglu, K.: WaveNet: A generative model for raw audio (2016). ArXiv:1609.03499v2
Wang, C., and Dubnov, S., Pattern Discovery from Audio Recordings by Variable Markov Oracle: A Music
Information Dynamics Approach, Proc. of 40th IEEE International Conference on Acoustics, Speech andSignal Processing (ICASSP), Brisbane, 2015
Wang C. and Dubnov, S., The Variable Markov Oracle: Algorithms for Human Gesture Applications, IEEE
Multimedia, vol. 22, no. 4, pp. 52 – 67, 2015
Wang C., Hsu, J., and Dubnov, S., Machine Improvisation with Variable Markov Oracle: Toward Guided andStructured Improvisation, ACM Computers in Entertainment Journal, Second Special Issue on Music
Meta-creation, 2016
Wiggins, G.A., Searching for computational creativity. New Generation Computing, 24(3):209–222, 2006.
Yang, L.C., Chou, S.Y., Yang, Y.H.: MidiNet: A convolutional generative adversarial network for symbolicdomain music generation. In: Proc. of the 18th International Society for Music Information Retrieval
Conference (ISMIR 2017). Suzhou, China (2017)13

AssayagPart B1

REACH

Section b: Curriculum vitae
PERSONAL INFORMATION
Family name, First name: Assayag, Gérard, Date of birth: 20/06/1960, Nationality: French
URL for web site: repmus.ircam.fr/assayag•

EDUCATION

2009
1985
1977-1984•

CURRENT POSITION(S)1992 – 2019•2003 – 2007
1986 – 1992
1990-1991

UCSD Distinguished Scholar grant recipient, invited researcher UCSD, San Diego, USA JSPS grant recipient, invited researcher at Tokyo University Todai.
Erasmus Mundus Team 2018 grant recipient, invited researcher at Tokyo University.
Best Presentation Award, International Computer Music Conference 2007
Winner Metro Company (RATP) Design Contest for the Centenary of the Parisian Metro
Winner of the "Micro-Computer" and the "Art and the Computer" competitions launched by
the French Government.

2018
2007
2000
1981

SUPERVISION OF GRADUATE STUDENTS AND POSTDOCTORAL FELLOWS

1992 – 2017
•

2019
2017
2015
2014
2012
2011
2009

I have directly supervised 22 PhD, 23 Masters, 12 post-docs, at STMS Lab with Université
Pierre et Marie Curie (UPMC), then Sorbonne University.

TEACHING ACTIVITIES (if applicable)

1991 – 2007•

Head of the laboratory Sciences and Technologies of Music and Sound (STMS Lab)STMS is a 125 people joint lab co-operated by by IRCAM, CNRS, Sorbonne University
Coordinator, ATIAM Master’s program (Acoustics, Signal Processing, Computer
Science for Music), Sorbonne University and 4 partner universities.
Free-lance consultant R&D in digital intelligence systems for sound and music
Professor of computer science for Musical research and creation at IRCAM.

FELLOWSHIPS AND AWARDS

2019

•

Head of Music Representations Research Group at STMS Lab (IRCAM, CNRS, Sorbonne
University). Team’s production : +50 PhD thesis, +100 Master thesis, + 150 Journal papers,
300 proceedings papers, + 100 books and chapters, 15 research awards

PREVIOUS POSITIONS

2011 – 2017•

Habilitation à diriger les recherches in Computer Science.
Université de Bordeaux II, Ecole doctorale de mathématique et d'informatique, France
Master in Computer Science
Université Pierre et Marie Curie (UPMC), Paris
Undergraduate and graduate studies in Music, Musicology, Philosophy, Literature, Linguistics,
Mathematics and Computer Science.

Musical Informatics, Master ATIAM, Faculty of Engineeeing, Université Pierre et Marie
Curie, IRCAM, — Computer Science for musicians, IRCAM

ORGANISATION OF SCIENTIFIC MEETINGS. Leading organiser of :ImproTech Paris-Athina 2019, international workshop and festival on co-improvisation with
Digital Intelligence, Athens, Greece, September 2019.
Improtech Paris -Philly 2017, Philadelphia, Dec 2019.
Mathemusical Conversations. International workshop, Institute for Mathematical Sciences,
Singapour. Montreux Jazz Festival International Workshop : Musician and machine.
ELS'14 - 7th European Lisp Symposium, IRCAM, Paris, France
Improtech Paris -New-York 2012, New-York, May 2012.
MCM 2011 International Conference on Mathematics and Computation in Music, IRCAM,
Paris. Forum 2011 «Next Generation of Virtual Reality», Todai Tokyo University.
International Symposium on Complexity in the arts and sciences, IRCAM, Centre Pompidou
14

Assayag
2004
2000
1999

INSTITUTIONAL RESPONSIBILITIES (if applicable)

2019
2011-20192015-20172011-201720132012
2003-2007•REACH

SMC 2004, First International Sound and Music Computing conference, IRCAM, France.
MaMuPhi, monthly seminar in Mathématics, Music and Philosophy, Ircam, 2000/2001.`
Forum Diderot Mathematics and Music, international workshop under the auspices of theEuropean Mathematical Society.
International Symposium on New Technologies for Music Printing, MIDEM, CANNES.

1987•

Part B1

Member of the HCERES (High Authority for Research Structures Evaluation) evaluation
expert board, founding member Sorbonne Center for Artificial Intelligence (SCAI)
Member of Sorbonne Université (SU) Doctorate School selection committee for PhD Grants
Founding Member : Collegium Musicae research Institute, Sorbonne Université,
Head STMS Lab, IRCAM, CNRS, Sorbonne University, Member of the council and bureau of
SU doctoral school, Member of the Scientific Council, Faculty of Engineering, SU.
Member of the Evaluation Expert Board, Agence Nationale de la Recherche (ANR), program"Contenu et Interaction" 2010-2013
Member of the Experts Committee, evaluation of European Project Integra II, Fusing Musicand technology, EU Culture Program
Head ATIAM Master’s Program, Pierre et Marie Curie University

MEMBERSHIPS OF SCIENTIFIC SOCIETIES (if applicable)

Founding Member, Society for Mathematics and Computation in Music, Founding Member, French Society
for Musical Informatics, Member FWO Research Society on Foundations of Music Research.
•

MAJOR COLLABORATIONS (if applicable)

Shlomo Dubnov, CREL Lab, UCSD, USA, Marc Chemillier, CAMS Lab, EHESS, France, Elaine Chew,
C4DM Lab, Queen Mary, U.K., Philippe Codognet, Japan-French Lab of Informatics, Tokyo U., Japan,
Adrien Mamou-Mani, Hyvibe Sound Tech Startup Company, Tatsuya Harada, AI Lab, Tokyo U.
•

RESEARCH CONTRACTS

2020-2022
2018-2025
2018-2021
2017-20202014-20182013-2017
2009-20132012-20152010-20132010-20132010-2013.
2007-2010
2001-2003•
1998•

MERCI, National Research Agency (ANR) Grant, AI creative agents, Mixed Reality, PI.
ACTOR, SSHRC/NSERC Partnership Grant, Timbre and Orchestration Modeling
TENOR, Canadian SSHRC Partnership Development Grant, Music Notation and Digital
Intelligence
MAKIMOno, ANR international France/Canada Grant, Multivariate Analysis and Knowledge
Inference for Musical Orchestration
DYCI2, ANR Grant, Dynamics of Creative Improvised Interactions, PI
EFFICACE, ANR Grant, Dynamic Time Structures in Comp. Assist. Composition.
IMPROTECH, ANR Grant. Improvisation and Machines.
INEDIT ANR, Grant. Composing Interaction and Time.
SAMPLE ORCHESTRATOR 2, ANR Grant, New generation orchestration
CREMUSCULT, ANR grant, Modeling Music Creativity and Cultural Impact
REACT+, Colombia COLCIENCIAS Grant, Concurrent Constraint Programming
SAMPLE ORCHESTRATOR, ANR Grant, New generation sampling
CUIDADO, IST European Project, 5th PCRD, Music Information Retrieval

PATENTS
Process for sympathetic resonances on digital piano, Inventors: G. Assayag, G. BlochLEADERSHIP / selected examples of young scientists advancement

Some former PhD students and post-docs trained by G. Assayag have become : Professor (Sorbonne U., C.
Agon, U. of Brisbane A. Brown, U. of Cali, G.Sarria, U. of Nantes, C. Truchet, U. of Oslo, O. Lartillot, U.
Santa Cruz, P. Nauert), Researcher at Sony Computer Science Lab (P. Hanappe), award-winning musician (G.
Nouno, J. Blondeau, L. Morciano, L. Rokita, F. Avitabile, D. Ghisi), CEO in high-tech companies (A. Cont, F.
Maniatakos, B. Zamborlin, C. Laurier); Computer Music Designers (B. Lévy, B. Meudic); industry executives(O. Delerue, G. Carpentier)15

AssayagPart B1

REACH

Appendix: All ongoing and submitted grants and funding of the PI (Funding ID)
On-going Grants (Please indicate "No funding" when applicable):
Project
Title

MERCI

Fundi
ng
sourc
e
ANR

Amount(Euros)

Rol
e of
the
PI
PI

Period600K€

2020-2022

ACTOR

SSHR
C

1679K€

2018-2025

Exp
ert

TENOR

SSHR
C
ANR

24K€

2018-2021

500K€

2017-2021

Exp
ert
Exp
ert

Makimono

Relation to current
ERC proposal2
MERCI is mainly focused on industrial
collaboration and transfert of music
software technologies, a different focus
from REACH
This project with 18 international partners
focusesonassisted
Classical
Orchestration, a different focus fromREACH
TENOR concerns music notation with nooverlap with REACH.
As a difference with REACH, this projectis highly focused on data science tools fororchestration and composition

Grant applications (Please indicate "No funding" when applicable):
Project
Title
DemoCreaCy

FundingsourceANR

Amount(Euros)
488K€

Period2020-2023

16

Role of the PI
Chair

Relation to current
ERC proposal2
DemoCreacy is auniversity
chairoriented toward AI
educationandteaching,adifferent
focus
from REACH

AssayagPart B1

REACH

Section c: Ten years track-record
Gérard Assayag, an IRCAM senior researcher, has founded and currently heads the Music Representationteam (RepMus) in the STMS Lab. STMS (Sciences and Technologies of Music and Sound) is a joint research
unit co-operated by IRCAM, CNRS (Centre National de la Recherche Scientifique) and Sorbonne University.
IRCAM (Institut de recherche et de coordination acoustique / musique) created by famous conductor andcomposer Pierre Boulez is the world’s largest institution dedicated to both music creation and sound/musicscientific research. Assayag has been the Head of STMS Lab from 2011 to 2017 and, as such, involved innational and international research policies in Computational and Human Music Sciences, with a populationof 125 people (researchers, engineers, techs, admin and PhD students) in his jurisdiction.
Assayag is a co-founder of Collegium Musicae, the Sorbonne University Institute for Music Sciences ; hehas been instrumental in the recent creation of the Sorbonne Institute for Artificial Intelligence (SCAI)
where he has conceived the Digital Humanity Axis programme. He has co-founded the French Society for
Computer Music (AFIM), the international learned Society for Mathematics and Computation in Music(SMCM) and its associated first-of-kind peer-reviewed Journal of Mathematics and Music (JMM), now apillar of the research community in music computing.
Assayag’s personal researches (involving supervision of 22 PhD and 23 Masters works) and team direction(encompassing over 50 PhD works and 100 Masters) has resulted in several flagship technologies covering
Computer Assisted Composition, Orchestration or Improvisation, (OpenMusic: +100,000 downloads, OMax /Dyci2: + 100 concerts on international scenes and festival including Montreux Jazz Festival or Paris Centre
Pompidou, interaction with legendary musicians including David Bowie’s companion pianist Mike Garson orcontemporary jazz headliners Steve Coleman and Steve Lehman).
Assayag has shown strong leadership in the training and advancement of young scientists. The students andyoung researchers he has supervised have become professors in universities throughout the world, head ofhigh-tech companies or industry executives, award winning musicians (see CV). As head of the pluridisciplinary ATIAM masters, G. Assayag has defined its pedagogical program in the frame of the “Bologna
Process” to ensure world-wide comparability in the standards and quality of higher-education qualifications.
More than 150 ATIAM students have since engaged in a PhD, of which the greatest part have thereafter
become university professors, researchers, industry executives or high-tech startup leaders all around Europe.
Assayag has defined through theoretical publications and popular technologies the concept of symbolic
interaction to account for rich and versatile musical dialog between machines and humans, traversing several
levels and scales of information, from acoustic signal to higher symbolic and cognitive structures. This concept
is now evolving towards general co-creativity in cyber-human workship, a central theme of this project.

Selected Publications
Déguernel, K., Vincent, E., Assayag, G., Probabilistic Factor Oracles for Multidimensional Machine
Improvisation. Computer Music Journal, 42:2, article 4, 2018
Nika, J., Chemillier, M., Assayag, G., ImproteK: introducing scenarios into human-computer music
improvisation, ACM Computers in Entertainment, 14 : 2, article 4, 2017.
Assayag, G., Improvising in Creative Symbolic Interaction. In Smith; Chew; Assayag (Eds), Mathematical
Conversations: Mathematics and Computation in Music Performance and Composition, World Scientific;
Imperial College Press, pp.61 - 74, 2016
Smith, J., Chew, E., Assayag, G., (Eds), Mathemusical Conversations : Mathematics and Computation inMusic Performance and Composition. World Scientific; Imperial College Press, 2016.
Assayag, G. Creative Symbolic Interaction, 40th Intl. Comp. Mus. Conf. and 11th Sound and Music Computing
Conf. (ICMC / SMC joint conf.), Athenes, Greece, pp 1-6, 2014.
Dubnov, S., Assayag, G., Music Design with Audio Oracle using Information Rate. First International
Workshop On Musical Metacreation (Mume 2012), Oct 2012, Palo Alto, United States. pp.1-1, 2012.
Cont, A., Dubnov, S., Assayag, G., On the Information Geometry of Audio Streams with Applications to
Similarity Computing, IEEE Trans. on Audio, Speech and Language Processing, 19 (4), pp.837-846, 2011
Dubnov, S., Assayag, G., Cont, A., Audio Oracle Analysis of Musical Information Rate, Proceedings of IEEE
Semantic Computing Conference (ICSC2011), United States. pp.567—571, 2011
Assayag, G., Bloch, G., Cont, A., Dubnov, S., Interaction with Machine Improvisation, in S.Argamon, S.
Dubnov and K.Burns (Eds) The Structure of Style: Algorithmic Approaches to Understanding Mannerand Meaning, 2010, Springer, pp.219-245
17

AssayagPart B1REACH

Assayag, G., Gerzso, A., New Computational Paradigms for Computer Music , Delatour, Paris, 2009

Selected Invitations
G. Assayag has delivered invited talks (distinguished scholar) or keynote-talks at :Collège de France, Ecole Normale Supérieure, Radcliffe Institute Harvard, Ecole Polytechnique, Ecole
Centrale, Ecole des Mines, Centre Pompidou, Boston University, U.C. San Diego, U. of Southern California,
U.C. Santa Barbara, CIRMMT Mc Gill U., Todai U. Tokyo, Geidai U. Tokyo, Tohoku U. Sendai, U. ofCampinas, NYU Abu Dhabi, Montreux Jazz Festival, Onassis Cultural Center Athens, Pontificiad Javeriana
Universidad de Cali, Queen Mary U., U. of Athens, Institute for Mathematical Sciences NUS Singapore,
Lorentz Center Leiden, International Computer Music Conference, Sound and Music Computing, European
Lisp Symposium Madrid, Intl. Conf. on Inf. Elect. Systems Sendai, U. Pompeu Fabra Barcelona, Universityof Minnesota, Calit2 Institute UCSD, Tokyo NTT InterCommunication Center.

Board Memberships
G. Assayag serves or has served recently on:
Advisory Board of the Springer book series Computational Music Science, Editorial board of Journal of
Mathematics and Music : Mathematical and Computational Approaches to Music Theory, Analysis,
Composition and Performance (founding member), board of Society for Mathematics and Computation inMusic (founding member), Editorial Board of the Journal of New Music Research, board of directors of theFrench Musical Informatics Association (founding member), executive board of Sorbonne Collegium Musicae(founding member), scientific advisor board Sorbonne Center for Artificial Intelligence, reviewing board of
HCERES (National High Authority for Research Structures Evaluation), Scientific Council of Sorbonne
University Science Faculty, Executive Board of Sorbonne University Doctoral School EDITE, reviewing
board of EU Culture Program, reviewing Board of Agence Nationale de la Recherche (ANR)

Conference organization
G. Assayag has been a leading organizer of international workshops or symposia :ImproTech Paris-Athina 2019, international workshop / festival on cyber-human co-improvisation, Athens Sep
2019 ; Improtech Paris-Philly 2017, Philadelphia, Dec 2019 ; Mathemusical Conversations Intl. Workshop,
Inst. for Math. Sciences, NUS Singapore, Jan 2015 ; Montreux Jazz Festival Intl. Workshop : Musician andmachine, Jul. 2015, ELS'14 - 7th European Lisp Symposium, IRCAM, Paris, 2014 ; Improtech Paris -NewYork 2012, New-York, May 2012 ; MCM 2011 Intl. Conf. on Mathematics and Computation in Music,
IRCAM, Paris 2012; Forum 2011 «Next Generation of Virtual Reality», at Tokyo and Tohoku University ;International Symposium on Complexity in the arts and sciences, IRCAM, Centre Pompidou, 2009.

Creative activity
G. Assayag has performed a great amount of concerts in high visibility venues, with world class performers,
using or demonstrating the results of his researches in computational co-creativity including :The Loft, UCSD, Uzeste Music Festival, Annenberg Center Philadelphia, Centre Georges Pompidou Paris,
IRCAM, Collège de France l, Montreux Jazz Festival, Singapore Yong Siew Toh concert Hall, New York
University @ Abu Dhabi Concert Hall, Cité des sciences et de l’industrie Paris, Béla Bartók National Concert
Hall Budapest, The Roulette Brooklyn, Conservatoire de Paris, Musée des confluences Bordeaux, Hexagone
Concert Hall Meylan, Music Instruments Museum Berlin, Molière Concert Hall Bordeaux, Le Rocher Palmer
Concert Hall Bordeaux.

18