RMCP: Remote Music Control Protocol
Design and Applications
Masataka Goto

Ryo Neyama

Yoichi Muraoka

School of Science and Engineering, Waseda University
3-4-1 Ohkubo Shinjuku-ku, Tokyo 169, JAPAN.
fgoto, ryo, muraokag@muraoka.info.waseda.ac.jp
Abstract
This paper describes the design and various applications of a communication protocol in a distributed
music system that integrates MIDI and computer networks. We designed this protocol, called RMCP
(Remote Music Control Protocol), to share symbolized musical information through the networks. Most
previous related protocols were connection-oriented and did not emphasize efficient information sharing
among processes. Since the RMCP is a UDP/IP-based connection-less protocol, it supports broadcastbased information sharing without the overhead of multiple transmission. To enable real-time MIDI
handling, it also supports time scheduling using time stamps. RMCP has been used in various applications
such as a music-controlled virtual dancer, a virtual jazz session system, and a remote session system.

1 Introduction
Computer-network protocols for symbolized musical information such as MIDI information have been taking onimportance with the popularity of network computing because of various applications. Integrating MIDI and computer networks such as LANs (local area networks) and
WANs (wide area networks), for example, enables network musical applications such as live MIDI broadcasts
and remote sessions via network, computer supported
sessions such as human-computer improvisation withvarious interactive-graphics displays, and distributed implementation of music-related software to achieve goodload-balancing and to exploit various facilities connectedwith different computers. Such applications require efficient information sharing over the network.
Most previous MIDI-based network protocols [1,2, 3, 9] were connection-oriented and did not emphasize low-latency information sharing among multiple distributed processes. MIDI is also weak in efficient information sharing among multiple devices, although it
is somehow possible to achieve all-to-all communication
among those devices. In addition, its bandwidth is very
limited and it was designed just for local communication.
Although non-MIDI-based music protocols [8] have been
proposed, most of them presuppose the use of special devices.
In this paper we propose a music protocol, called
RMCP (Remote Music Control Protocol), which was
designed for sharing symbolized musical information
through computer networks. This is a communication
protocol in a distributed cooperative system that integrates MIDI and computer networks such as the Ethernet. Since it is a connection-less protocol on the UDP/IP,
it naturally supports broadcast-based information sharing
without the overhead of multiple transmission. Although
it was initially designed for use on a reliable LAN, it has

446

been extended to support WANs such as the Internet byusing a gateway program that can relay RMCP packets
over a WAN. It also supports time scheduling using a timestamp in each packet in order to enable real-time MIDI
handling.
Since 1992 we have developed not only the basic
MIDI I/O processes but also various RMCP-based applications such as (1) an interactive performance system
in which there is a music-controlled computer-graphics
dancer that two musicians can choreograph by their improvisation in real time; (2) a virtual jazz session system in which a human pianist can interplay with a virtual bassist and a virtual drummer who have computergraphics bodies for playing their instruments and making
gestures; (3) a networked session in which multiple musicians play together via the Ethernet with the assistance ofvisualization programs; (4) Improvision, with which untrained people can improvise music and interact with eachother; and (5) RemoteGIG, an innovative remote session
over the Internet that has a long delay, in which each musician improvises while listening to other players’ performances that are delayed for the constant period of a repetitive chord progression under a fixed tempo.
We have implemented RMCP programming libraries
in both C and Java languages and have used the protocol on various operating systems, such as IRIX-5.3,
IRIX-6.2, Solaris-2.5, SunOS-4.1.3, HP-UX, Windows95, Windows-NT, and Linux-2.0. RMCP has alreadybeen utilized for various purposes in several laboratories.

2 RMCPRMCP transmits various musical information in the formof RMCP packets, which are shared by multiple processeson distributed computers. For example, each MIDI message such as note-on and note-off is packed in a packet
and sent to other processes via a LAN.

PROCEEDINGS ICMC97

Table 1: The basic RMCP servers and clients.
RMCP Server
RMCP Sound Server
RMCP Display Server
RMCP Animation Server

RMCP Recorder

RMCP Client
RMCP MIDI Receiver
RMCP MIDI Station
RMCP SMF Player
RMCP Player

Description
send MIDI messages of received packets to a MIDI instrument
visualize MIDI messages of received
packets in the form of a piano keyboard
generate music-driven real-time computer graphics corresponding to received packets
record all received packets, with the received time stamps, in an RMCP Packet
Record File
Description
receive MIDI messages from MIDI
instruments
substitute a computer keyboard andmouse for a MIDI keyboard instrument
play a Standard MIDI Fileplay an RMCP Packet Record FileRMCP is based on the server-client model in which
multiple servers receive requests from various clients.
All RMCP-based processes are therefore categorized into
RMCP clients and RMCP servers, which can be considered input and output devices, respectively. An RMCP
client generates RMCP packets such as those including
MIDI messages received from a MIDI device (MIDI IN)
and including messages corresponding to user’s interactions, and it broadcasts the packets to all RMCP servers.
Since this broadcast is a connection-less one-directionaltransmission, there is no acknowledgement packet fromthe servers. Each RMCP server, on the other hand, receives all the broadcast packets and utilizes them in various ways, such as controlling MIDI devices (MIDI OUT)
for outputting sounds, visualizing musical information,and producing music-driven computer graphics. RMCP
servers can thus share all the broadcast packets without
the overhead of multiple transmission. Moreover, it isalso possible to add various servers such as a visualizing
server without any extra packets.

2.1 RMCP Servers and RMCP Clients
The design policy in developing RMCP servers andclients is to implement necessary functions as small different processes so that they can be reusable. This policy facilitates system implementation and expansion andenables those servers and clients to be allocated on distributed computers to achieve good load-balancing.
We have developed the basic servers and clients
listed in Table 1. Figure 1 shows an example of using
these servers and clients. A user plays on a MIDI instrument with an RMCP MIDI Receiver or on a computer
keyboard with an RMCP MIDI Station. When a MIDI instrument is used, MIDI messages from the instrument arereceived and broadcast as RMCP packets. When a computer keyboard is used, each key is assigned to a different
note (pitch) and the corresponding MIDI message is generated and broadcast. A user can also play a Standard
MIDI File using an RMCP SMF Player, which broad-

MIDI

RMCP MIDI Station
RMCP SMF Player

RMCP MIDI Receiver

Ethernet
RMCP Animation Server

RMCP Sound Server
MIDI

RMCP Display Server

Figure 1: An example of using RMCP servers and clients.

casts the contents of the file as RMCP packets. If RMCP
packets broadcast by these clients are received by RMCP
Sound Servers, included MIDI messages are sent to MIDI
instruments according to their time stamps as describedin Section 2.2. Those packets can also be visualized by
RMCP Display Servers in the form of a piano keyboard
and can be utilized simultaneously by various RMCP Animation Servers to generate music-driven real-time computer graphics such as virtual dancers and musicians.

2.2Time Scheduling

RMCP supports time scheduling using time stamps forreal-time packet processing. There are two kinds of
RMCP packets: those with a time stamp and those without a time stamp. The time-stamped packets received before their time stamps are kept in each server till the timeof their time stamps and are then processed on time andin order. This processing compensates for variations in
network latency. The time-stamped packets received after their time stamps and the packets without time stamps
are processed immediately.
This time scheduling requires time synchronization
among distributed computers because it is necessary for
RMCP servers on different computers to process packets
with the same time stamp at the same time. We therefore
introduced, on each computer, an RMCP Time Synchronization Server (RMCPtss) that enables all RMCP servers
to handle the time stamps of received packets as if all theinternal clocks of the different computers were synchronized.1 RMCP server developers thus need not care about
the time synchronization.
Each RMCPtss makes an offset table of temporal differences between its internal clock and the internal clocks
of other computers and periodically broadcasts a packet
including the table with the transmitted time stamp (itscurrent internal clock time). When an RMCPtss receives
the table from another RMCPtss, it updates the corresponding offset (temporal difference) in its own table bycalculating the difference between the transmitted timestamp of the received table and its received time measured
1 There are other solutions, such as NTP (Network Time Protocol)[RFC-1119] that synchronizes the internal clocks using administrative
authority. On the other hand, our solution does not require administrative authority because it does not directly adjust the internal clocks.

PROCEEDINGS ICMC97447

by its own internal clock.
Each RMCP server, on the other hand, receives thetable from the RMCPtss on the same computer. Everytime each server receives an RMCP packet with a timestamp, it adjusts the time stamp by using the sender’s offset in the table so that it can compensate for the internal
clocks’ difference.

3 Implementation of RMCPWe designed the communication protocol RMCP andhave implemented RMCP programming libraries and various servers and clients. Under the assumption that the
RMCP will be used on a reliable low-latency network
such as an Ethernet LAN, RMCP uses the UDP/IP (which
is an unreliable connection-less protocol) as its transport
protocol and does not ensure the reliability of communication except for remote communication over a WAN
such as the Internet.

3.1 RMCP Packets
Each RMCP packet consists of a message header andbody. The header specifies the message type, timestamp (with millisecond resolution), target server identifier, sender identifier, and message length. Some of thebasic message types are MIDI information for transmitting MIDI messages, beat information for beat synchronization, chord information for broadcasting chord nameand voicing, and animation information for controllingcomputer graphics.

3.2 RMCP over the Internet
Since RMCP was initially designed for use on a reliable
LAN, it was hard to use on an unreliable WAN such as theInternet. We have therefore introduced RMCP Gateways
that can provide the bidirectional relay of RMCP packets over the Internet by using the TCP/IP, which is a reliable connection-oriented protocol. A pair of RMCP Gateways connects two remote LANs: each gateway relays
the RMCP packets broadcast on its LAN to the gateway
on the other side of connection and also broadcasts thepackets relayed from the other gateway.2 RMCP servers
and clients can thus communicate as if the different LANs
connected by the gateways were the same network withthe exception of the network latency.
It is impossible to avoid the network latency over theInternet: even signals traveling at the speed of light needmore than 66 ms to reach halfway around the globe. We
therefore consider that the RMCP should provide a certain constant latency with very small deviation. A pair ofgateways measures and reports the network latency at thestart so that we can specify enough large constant latency
to avoid the latency deviation. The pair then punctually
relays the packets with the specified latency. This enables
live MIDI broadcasts and an original remote session RemoteGIG (described in Section 4.5).
2 Note that the relayed packets should be marked for the RMCP Gateways in order to avoid a possible infinite loop of the relay.

448

3.3

Experimental Results

We have implemented RMCP programming libraries inboth C and Java languages and have tested the protocol on various computers and operating systems, such as
IRIX-5.3, IRIX-6.2, Solaris-2.5, SunOS-4.1.3, HP-UX,
Windows-95, Windows-NT, and Linux-2.0. We foundthat RMCP facilitated the implementation and expansionof distributed music applications, especially real-time andinteractive applications.
We measured the communication delay when RMCP
packets traveled from a client to a server on different computers. We tested the C-based RMCP libraries on two
different SGI Indigo2s (IRIX-5.3) connected via the 10
Mbps Ethernet, The delay ranged from 0.28 to 1.24 ms,
and the average delay was 0.30 ms (standard deviation =0.06 ms). This result shows that RMCP is fast enoughcompared with the MIDI data transmission rate (31.25
Kbits/sec).

4 Applications
Various RMCP-based applications have been developed,
and this section introduces five of them.

4.1

Virtual Dancer “Cindy”

We have developed an interactive performance system inwhich two players, a drummer and a guitarist or pianist,
can together choreograph a virtual dancer called Cindy3by their musical improvisation in real time [7] (Figure 2).
This system enables the players to communicate using
both auditory and visual information. Because the players not only improvise together as in a conventional jam
session but also observe the virtual dancer whose motion
is changed according to their musical performance, they
can interact with each other not only through music but
also through 3-D computer animation.
This application consists of an RMCP Animation
Server for Cindy, an RMCP Music Analyzer (applicationspecific server) to analyze the guitarist’s improvisation,
an RMCP MIDI Receiver, and an RMCP Sound Server.

4.2

VirJa Session

We have proposed a virtual jazz session system called
VirJa Session in which a human player and computer
players can communicate not only by listening to eachothers’ performances but also by seeing each others’ bodies and gestures [5]. The human player sees the bodies and gestures of other computer players shown on 3-D
real-time computer graphics, and can feel their presence
as if they were actually playing together. In addition, eachcomputer player reacts to the gestures of the human player
by using a video camera. We can thus achieve multimodal
interaction among all players.
The current implementation of this system deals witha jazz piano trio consisting of a human pianist, a computer
3 Cindy can also dance in time to music by using our beat-tracking
system [6].

PROCEEDINGS ICMC97

1 period

1 period

1 period

time

Site A
Player A A-1

A-2

A-3

B-1

B-2

A-1

A-2

B-2

B-3

Site B
Player B B-1

Figure 2: Virtual Dancer “Cindy”.

Figure 4: RemoteGIG.

Figure 3: VirJa Session.

bassist, and a computer drummer (Figure 3). It consistsof an RMCP MIDI Receiver, an RMCP Sound Server,
and four kinds of application-specific RMCP servers andclients described in [5].

4.3 Networked session
The RMCP has been used in a local networked session inwhich several players perform music together via the Ethernet [4]. The players can not only listen to other players’
performances but can also see visualized performancesthat facilitate musical cooperation. This application requires RMCP MIDI Receivers or RMCP MIDI Stations,
RMCP Sound Servers, and RMCP Display Servers. An
RMCP SMF Player is useful for background accompaniment. An RMCP Recorder and an RMCP Player are alsoavailable for recording and playing back the performance.

4.4 Improvision
We have developed a musical-instrument interface, called
Improvision, which enables an untrained novice to improvise unconventional music easily by clicking and dragging a computer mouse [4]. Multiple players can interact with each other by playing RMCP Improvisions(application-specific clients) instead of RMCP MIDI
Stations and can see other players’ performances on
RMCP Improvision Display Servers (application-specificservers) instead of RMCP Display Servers.

4.5 RemoteGIG
RemoteGIG is an innovative remote session over the Internet that has a long delay. RemoteGIG overcomes the
network latency and offers a new possibility for future
remote sessions. It assumes that the tempo is constantand the chord progression is repetitive, like the 12-barblues chord progression. Figure 4 illustrates how remote
players interact with each other over the network. RemoteGIG turns the network latency to its advantage: eachplayer improvises while listening to other players’ performances that are delayed for the constant period of therepetitive chord progression. Because the progression isrepetitive, the delayed performances can fit the chords.
RemoteGIG requires RMCP Gateways in addition to theservers and clients described in Section 4.3. In particular,an RMCP SMF Player is needed for playing backgrounddrums to keep the constant tempo.

5 Conclusion
We have described a network protocol called RMCP thatenables multiple distributed processes to share symbolized musical information such as MIDI information. It
supports efficient broadcast-based information sharing
over a LAN and time-scheduling using time stamps toenable real-time MIDI processing, and it also enables
live MIDI transmission over a WAN such as the Internet. RMCP has been implemented on various operating
systems and has been utilized for various purposes.
We plan to distribute our RMCP software package on
our WWW page “http://www.info.waseda.ac.jp/muraoka/members/goto/RMCP/” and to provide an API to make
RMCP and MIDI usable in Java applets.

Acknowledgments
We thank the SALA (Science Art Laboratory), which enabled
the first author to originate this project. In particular, we thankYuji Hashimoto, Shigeru Chiba, and Shin Miyakawa for theirsupport in the early stages of the project. We also thank Yoshiaki Kikuchi, Isao Hidaka, Tetsuya Abe, Hideaki Matsumoto,
Yosuke Kuroda, Mitsukazu Washisaka, Keiji Hirata, and Yoichi
Nagashima for their helpful comments and cooperation.

References

0[1] T. Aoyagi and K. Hirata. Music server system distributed music
system on local area network . Journal of Information Processing, 15(1):1 9, 1992.
[2] D. Fober. Real-time Midi data flow on Ethernet and the software architecture of MidiShare. In Proc. of ICMC 1994, pages 447 450,
1994.
[3] D. Fober, S. Letz, and Y. Orlarey. Recent developments of
MidiShare. In Proc. of ICMC 1996, pages 40 42, 1996.
[4] M. Goto and Y. Hashimoto. A distributed cooperative system toplay MIDI instruments toward a remote session (in Japanese).
IPSJ SIG Notes, 93(109):1 8, 1993.
[5] M. Goto, I. Hidaka, H. Matsumoto, Y. Kuroda, and Y. Muraoka. A
jazz session system for interplay among all players VirJa Session. In Proc. of ICMC 1996, pages 346 349, 1996.
[6] M. Goto and Y. Muraoka. Beat tracking based on multiple-agent
architecture a real-time beat tracking system for audio signals. In Proc. of ICMAS 1996, pages 103 110, 1996.
[7] M. Goto and Y. Muraoka. A virtual dancer "Cindy" interactive
performance of a music-controlled CG dancer . In Proc. of theLifelike Computer Characters ’96, page 65, 1996.
[8] K. McMillen, D. Simon, and M. Wright. A summary of the ZIPI
network. CMJ, 18(4):74 80, 1994.
[9] O. Nielsen. MIDI and audio via ISDN. In Proc. of ICMC 1994,
pages 451 454, 1994.

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

PROCEEDINGS ICMC97449