Asynchronous Migration of Execution Context
in Java Virtual Machines
Kazuyuki Shudo∗, Yoichi Muraoka
School of Science and Engineering, Waseda University,
3–4–1 Okubo, Shinjuku-ku, Tokyo 169–8555, Japan

Abstract
The migration of the execution context has been applied to remote execution andmobile agents, and asynchronous migration can be applied to even more applications,such as load balancing. We have therefore designed a system for the migration of Java
threads, one that allows asynchronous and heterogeneous migration of the execution
context of the running code. This paper describes an overview of the system, theproblems we have faced in designing its facilities, and the results of preliminary
evaluations of it performance.
Key words: Thread migration; Execution context; Asynchronous migration;
Runtime system; Java Virtual Machine1

Introduction

Correspoinding
author.
E-mail:shudoh@muraoka.info.waseda.ac.jp;
tel.:+81 3 3209 5198; fax:+81 3 3209
5198.

We are designing a migration system for Java threads, and it will
provide all three of these functions.
The system, called MOBA, supports
the asynchronous migration of Java
threads in heterogeneous various
platforms but the facility for execution of native code has not been
implemented yet. The system allows the migration of an execution
context to be ordered using simple
programming and user interfaces.
Programmers can instruct a thread
to migrate with one statement:“goTo(destination)”. A migrant in
the system is a thread with its execution context. An execution context consists of the states of methods being executed, and it resides in

Preprint submitted to Elsevier Preprint

22 January 2000

The migration of execution context,
known as process and thread migration, has long been applied toload balancing and remote execution[4][8][3], and has recently been
applied to mobile agents[18][14][13].
One of the challenges remaining inthis area is to design facilities that
support asynchronous and heterogeneous migration, as well as theexecution of native code.
∗stack frames that contain the program counter, operands, and local
variables. The context can be saved
and moved to other Java virtual machine(JVM)s without cooperation of
the running code, and its execution
can be resumed.

“Asynchronous” means that the migration can be carried out without
the awareness of the running code.
Asynchronous migration allows entities outside of the migrating thread
to give the order to migrate. Usersand threads other than the migrant
can issue orders. Accordingly, anappropriate program, such as a job
scheduler, can attempt to balanceloads of JVMs with migration.

Heterogeneity of machines, such as
diﬀerences in processors and operating systems, surely complicate
the migration approach. Our system
make heterogeneous migration[17]possible by handling the execution
context in the JVM rather than ina particular processor or in an operating system. Threads in our system
can thus migrate between JVMs on
diﬀerent platforms.

In the rest of this paper, we describe
an overview of the designed systemand discuss design and implement
issues of thread migration facilities.
In the section 2, we try to compare
MOBA with some existent systems
and methods. In the section 5, weshow the results of preliminary performance evaluations.

2

Related Work

The properties of systems supporting
the migration of execution context
and mobile agents are listed in Table
1. The entry in the context column
shows whether or not the system
supports migration of execution context, that in the execution column
shows how the code is executed, thatin the heterogeneity column showswhether or not the system allows migration between machines diﬀeringin processor and underlying OS, andthat in the asynchronousness column
shows whether or not the systemallows asynchronous migration.
Sumatra[12] and TeleScript[18] are
the systems most similar to MOBA.
They diﬀer in programming language — TeleScript has its own language, whereas MOBA and Sumatra
adopt Java — but their mechanisms
are similar. The code is executedby an interpreter, and the execution
context can migrate. Only MOBA,
however, provides asynchronous migration. Migration can be carriedout without awareness of the running code. In other words, orders
to migrate can be issued by entities
other than the running code. Thismeans that the timing of migration
can be determined at runtime. Although the mechanism of TeleScript
doesn’t prevent asynchronous migration, TeleScript does not oﬀer
programming or user interfaces for
it. And Sumatra allows only explicit
migration using a go() method.
JavaGO[14][13] and Fünfrocken’s
method[5] also can save and move theSystemLanguage

Context

Execution

Heterogeneity

Asyncronousness

MOBA

Java

yes

interpreter

yes

yes

SumatraJava

yesinterpreter

yes

no

JavaGOJavayes

JIT or interpreter

yes

no

Voyager,Aglets

Java

no

JIT or interpreter

yes

no

TeleScript

TeleScriptyesinterpreter

yes

no†

Emerald

Emerald

yes

native code

no

yes

ArachneC, C++

yesnative code

yes

no

† The mechanism does not prevent asynchronous migration. But the way to issue
an order from the outside of the running code is not provided.
Table 1
Properties of various migration systems.

execution context of a Java program
and restore it on another machine.
Their approach is based on preprocess or source code translation.
Arachne[3], which is a thread system
for the C and C++ languages, is also
based on the sort of method. Theirmethod applied to Java has the advantage of being able to work with
fast existing JIT compilers. Dedicated JVMs, either that are extended
or built from scratch, can not beneﬁt from existing JIT compilers. Oneof the problems of this translation
approach is that its area of application is more limited than that of aruntime-system approach. Because
the translation approach requires thetiming of migration to be describedin migratory codes, it cannot be applied to some of the applications thatneed asynchronous migration, suchas load balancing.

3

Overview of the System andthe Scheme

MOBA is implemented as a plug-in
to the JVM, that is implemented
by Sun Microsystems and dealt out
distributed as Java Development
Kit(JDK) and Java Runtime Environment(JRE). The most part of
MOBA is written in Java. Althoughsome of its code is in C language,
the system supports any UNIX platform where Sun’s JVM can run.
MOBA is a plug-in, not a JVM built
from scratch, so a program utilizing MOBA functions can also utilize
plenty of functions provided by the
JDK.

3.1 Programming and User Interface

The programming interface provided
by MOBA is so simple that only a
few changes to the original code areneeded to make the code migratory.

To make the thread movable, we use
the MobaThread class instead of thenormal Thread class to instantiate
the thread. To migrate to anothermachine, call the following method.
MobaThread.goTo(destination)

User Application
ThreadMigrationis MOBA.

ThreadExternalization

Accesswithnative methods(in C lang.)

ObjectMarshaling
Introspection

Unlike programmers working withexistent mobile agent systems forJava[9][11][6], programmers working
with MOBA have to pay little attention to the particular programming
interface.

Java Virtual Machine

Fig. 2. Organization of thread migration facilities.
Thread
1. Attributes

Migration can be ordered not onlyby the migrant but also entities outside of the migrant, such as otherthreads and users. In this case, nostatement to migrate is required in
the migrant’s code. Other threads inthe same JVM, where the migrantstays, call the following method tomove the thread.
<target thread>.
moveTo(destination)
Furthermore, users can issue the order to migrate by using some userinterfaces, either character-based orgraphical(Fig. 1).

3.2

Organization of the Facilities

The migration facilities MOBA consist of some libraries, introspection,
object marshaling, thread externalization, and thread migration. Theirrelation and dependency are shownin Fig. 2. The introspection libraryprovides the same function as the reﬂection library which is part of thestandard library of Java. Similarly,

2. Reachable objects
from the thread

name, priority, ...
3. Stack frames

orderClass and method namePC to return (in offset)
Operand stack top
Last-executed PC
Local variables
Stack

Fig. 3. Procedure to externalize athread.

object marshaling provides the function of serialization. Thread externalization translates a state of the running thread to a byte stream, and it isused by the thread migration libraryfor moving threads between JVMs.
Thread externalization can be used
not only for migration, but also for
persistence and fault tolerance. We,or an appropriate daemon program,can save states of running threads toa disk or a database in order to provide for an unforeseen fault of an underlying OS or machine.
The procedure to translate a thread
to a byte stream is represented inFig. 3. First of all, some attributesof the thread (name, priority and soon) are translated. Then, after all

Fig. 1. Graphical user interface to visualize mobile threads and order to migrate.

the objects that are reachable fromthe thread object are marshaled, execution context is treated. A context
consists of contents of stack frames
generated by a chain of method invocations. The externalizer follows
the chain from older frame to newer
one and serializes the contents of theframe. A frame is located on the stack
in a JVM and contains the state ofa called method, The state consistsof a program counter, operands tothe method, local variables, elementson the stack. They are serialized inmachine-independent form.

4

Design Issues of Thread Migration in JVMs

This section addresses some issuesof implementing migration systemof execution context in JVM. Asynchronous migration is a type of migration ordered by the thread otherthan the migratory thread’s self. It
needs support by JVM and JIT. JIT
compiler have to provide a machineindependent form of execution context, or it cannot work with a runtime system such as MOBA. Common JIT compilers do not provide.
MOBA copies all the objects reachable from the migratory thread tothe migration target, but in somecases, selection of of objects is desirable. If the selection is possible,
MOBA can leave objects tied to the
local resources such as ﬁle and socket
descriptors. Current MOBA implementation disables those objects at
migration.

4.1 Asynchronous Migration“Asynchronous migration” is migration without awareness by the
migrant. The migration in MOBA
does not need any cooperation of the
migrant program. In contrast to it,synchronous migration is invoked by
the migrant program itself. This issuitable for some applications (e.g.,
describing mobile agents) but notfor other applications. Dynamic loadbalancing and saving running states
for fault tolerance are two applications that need asynchronous migration.
Migration of execution context needs
that the migratory thread is suspended at a migration safe point. It
is deﬁned as a point in the executionof JVM, at which point the JVM
is in a consistent state. A JVM canbe inconsistent in the middle of theexecution of a bytecode instruction,in other words, elements of the execution context (such as a program
counter, a stack pointer and so on)
are inconsistent.
MOBA allows asynchronous migration, but it requires nonpreemptive scheduling of Java threads. If
scheduling is preemptive, generally
threads can be suspended at a notsafe point. Nonpreemptive scheduling, on the other hand ensures thatthe suspended thread is at a safe
point. In nonpreemptive scheduling,
the thread suspends its own execution by calling methods to wakeup the scheduler. For example, theThread.yield() method, and someother methods that call system callsof the OS, kick the scheduler.
Sun’s JVM can utilize two kindsof libraries as an underlying thread
library: OS native threads (e.g.,
Solaris native threads) and green
threads. Scheduling in green threads
is nonpreemptive, so it always allows

asynchronous migration. Althoughwith preemptive scheduler, MOBA
can carry out cooperative migration(e.g., goTo(destination)).
There are some techniques to stopthreads at one of migration safe
points even with preemptive scheduling. Polling and code patching arewidely known. Patching is similar
to debuggers’ use of break points tosuspend execution at a given location. ResearchVM, which is a JVM
developed in Sun Microsystems Laboratories and is formerly called ExactVM or EVM, uses patching tosuspend threads at GC points[2].
Stack frame maps which shows thetypes of stack contents is provided
for each GC point, so the JVM needs
to stop threads at the points. If wemake a JVM and a JIT compiler byourselves, we can apply these techniques. Or ResearchVM is possibly
suitable for the base to implement
thread migration. Some features of
the JVM can support thread migration. GC points enables thread suspension at safe points because GC
points is also migration safe points.
Stack frame maps can eliminate type
inference of values in the stack (section 4.5).

4.2 Runtime Compilation

Most JVMs have a runtime compiler
called a Just-in-time compiler (JIT).
It translates bytecode to processor
native code at runtime. A runtime
system such as MOBA that supports
the capturing of execution context,
however, is incompatible with exist-

ing JIT compilers. Because the approach of MOBA and Sumatra[12]provides a runtime system supporting execution-context capture, theycannot work with existing JIT compilers.
Heterogeneous migration needs amachine-independent representationof execution context, but most existing JIT compilers don’t preserve a
program counter on bytecode. Only
the counter on native code can beobtained during execution of the native code generated by an existing
JIT compiler. But, Sun’s HotSpot
VM[16] may allow the execution
context on bytecode to be captured
during the execution of the generated native code. Its details is not
documented, but capturing the program counter on bytecode seems tobe needed for its dynamic deoptimization. Common JIT compilers do
not allow it.
The approach based on source codetranslation[14][5] may work with any
existing JIT compiler and thereby
beneﬁt from the JIT. But as noted
earlier, this approach requires thetiming of migration to be determinedwhen writing codes, it cannot be applied to areas such as load balancing
and fault tolerance.
Although none of the JIT compilers
presently available can work with theruntime system, but it should in principle be possible to design a JIT compiler that supports the capture of execution context.
we are now developing the sort of JIT
compiler: the native code sometimes

checks a ﬂag during its execution,and the ﬂag indicates a request for
capturing the context. This polling
may have some cost in term of performance, but we expect any decreasein performance to be small.

4.3 Selection of Objects to be Transferred

All the objects reachable from thethread object are marshaled andreplicated on the destination of the
migration. This may be a problem. In
the case that a object is transferredto the other machine and transferredto the original machine again, theoriginal object and the object transferred twice are diﬀerent objects. The
semantic of the program may change
if multiple threads share the object
and part of them migrate. And sometransferred objects will not be usedon the destination. Transferring all
the objects is wasteful in respect of
migration cost.
Selective migration may be able tosolve these problem. But there aresome problems to implement it. We
must develop an algorithm to determine which of the objects shouldbe transferred. And migration system have to cooperate with a kindof distributed object system whichenables remote reference and remote
operation. The migrated thread has
to access to remained objects withthe distributed object system. The
distributed object system have to be
touch with JVM closely. It must support interchange of a local reference
and a remote reference. And if it does

not allow some type of operations
such as array access, the migrated
thread which does them cannot work
properly.

kinds of resources, such as a ﬁle descriptor. Consequently, the ﬁle descriptor becomes invalid (e.g., −1)
at migration.

There are no existent systems thatsatisfy these requirements, so we havedeveloped a distributed object system supported by the JIT compiler
shuJIT[15] and we are integrating itwith MOBA. The system implements
all requirements mentioned above because it is supported by the JIT andable to cooperate with JVM closely.

Classes may be grouped into somecategories with regard to their dependence on resources. Some classes
whose instances are tied to the resources must be maintained, butothers do not have to be maintained.
When classes are newly written byprogrammers, the programmers canwrite speciﬁc marshaling methods
for them. But, if the classes residein the Java standard library, they
are not aware of the migration of
their instances. If the object migration system does not make objects
stationary, it has to treat the resources tied to the classes. Current
MOBA maintains some classes suchas FileDescriptor.

4.4

Marshaling Objects Tied to the
Resources

MOBA does not oﬀer a function handling objects that reside in a remotemachine, and it thus cannot by itself
handle a reference to a remote object. Objects reached from the migratory thread are copied to the migration target. How to maintain objects
which have some relation to resources
speciﬁc to the machine is a common
problem in object migration systems.
File and socket descriptors are examples of the resources.
The general solution for the system to support remote reference ismaking the objects ﬁxed to the machine[8][12]. Although integrationof MOBA with a distributed object
system (section 4.3) should makethe solution possible to adopt, current MOBA cannot use that solution because so far it cannot handle
remote references. To prevent accidents caused by an attempt to movethe resources, MOBA disables some

4.5 Types of Values on the JVMStack

The runtime system that supports
the capture of execution context has
to know the types of values in thestack of a JVM. Local variables andoperands of the called method stayon the stack. The values may be
32-bit or 64-bit immediate values orreferences to objects.
It is diﬃcult to distinguish the types
referring only the value. A Sumatrainterpreter maintains a type stackparallel to the value stack[1], anddistinguishes the type with it. Sumatra has its own interpreter built fromscratch, so it can use this method.

But MOBA is a plug-in to the existing Sun’s JVM, which does nothave a type stack like Sumatra. If theclass ﬁle has LocalVariableTable
attributes[10], the types of local variables can be obtained in the table.
But in general, in a Sun’s JVM there
is no information about the type ofvalues in the stack.
With a JVM like Sun’s, we have either to infer the type from the value
else determine the type by data ﬂow
analysis which trace the bytecodeof the method like a bytecode veriﬁer. Tracing bytecode to determine
types is computationally expensive,
so MOBA infers the type from thevalue. It distinguishes a reference
from an immediate value by utilizing the fact that all references residein the speciﬁc heap space and arealigned in 64-bit boundaries. Whena value is given, if it is in the memory area and aligned in the bound,the value is regarded as a candidate
for a reference. Furthermore, MOBA
checks the structure of the candidate. The value is considered as an
immediate value if it does not havethe right structure as a reference. According to this method, a reference
can never be mistaken for an immediate value, but it is possible thatan immediate value is infrequently
mistaken for a reference.
This inference and validation method
cannot be perfect. Although the possibility of misidentiﬁcation is nothigh, this is one of important problems which current implementation
has and should be solved. It can bea solution to make a stack frame
maps[2], or MOBA will be able torefer the maps if a JVM and a JIT
compiler provide them. The JVM
which MOBA runs on does not provide them, but ResearchVM does
(section 4.1).

5

Performance Evaluation

We evaluated the performance of
MOBA’s mobility function by using two machines connected via one
Ethernet repeater, in a 100-Mbit/sec
Ethernet. One of the machines had
an UltraSPARC–II 167-MHz processor, the other had an UltraSPARC–
II 296-MHz processor, and SunOS 5
ran on both machines. We used thereference implementation of JDK
1.1.8 with MOBA and used the production release of JDK 1.1.7 withother systems, and we used interpreter with MOBA and Sun JIT
compiler with other systems since
MOBA can’t work with existing JIT
compilers.

5.1 Latency of Migration

We described a simple and lightweight migrant with MOBA and with
Voyager ORB 3.0[11], and then madethem go and return. With MOBA
the following code is deployed in the
migratory thread’s self:
get the start time;for (i = 0; i < repeat time; i++) {go to the destination;
return to the original machine;}
get the end time;# of roundtrips

1

10

20

50

191.0

109.3

105.53105.32

Voyager
292.5 57.05
Table 2
Latency of an one-way migration (msec).

44.00

37.08

MOBA

In the case of Voyager, we provided
the following code outside the migratory object. The code issues instructions to migrate using Voyager’s mobility facilities.
create a migratory object;
get the start time;for (i = 0; i < repeat time; i++) {move the object
to the destination;
move the object
to the original machine;}
get the end time;

Migration times obtained with thesystems are listed in Table 2. It is interesting whether thread migration
can be comparable with the mobile
agent system like Voyager, which
does not support migration of execution context. Voyager shows lower
latency for multiple roundtrips. But
when a go and back is performed
once, MOBA is faster even though it
moves the execution context in addition to the data held by the migrant.
The initial use of Voyager’s mobility
seems to cost a lot. In such cases,
the cost of transferring execution
context will be acceptable.

5.2

Throughput

We also used MOBA for remote execution, measured the data transfer

throughput, and compared it withthe throughput obtained using two
object request broker(ORB)s for
Java, RMI[19] and HORB[7]. With
ORBs we used a remote method invocation with an argument and noreturn value. The argument was alarge array of 64-bit ﬂoating pointvalue:// preparation
double[] argument =new double[array size];
get the remote reference
into the variable remote ref;// measurement
get the start time;
remote ref.aMethod(argument);// remote invocation
get the end time;

In the case of MOBA, remote execution is done by thread migration emulating remote method invocation.
The migrant goes to the target machine with an argument and return.
the following code is deployed in the
migratory thread’s self:// preparation
double[] argument =new double[array size];// measurement
get the start time;
go to the target machine;argument = null;// this discards the argument
return to the original machine;
get the end time;Round trip time of remote execution
Round trip time (msec)

1600
RMI: 303n + 3.57

MOBA
RMI
HORB

1400
1200
1000

MOBA: 188n + 240

800
600

HORB: 257n + 4.99
400
200
0
0

1

2

3

4

5

Transferred data size (Mbyte)

Fig. 4. Round-trip time of remote execution.

As shown in Fig. 4, when the amountof data transferred is small MOBA
takes more time than the ORBs do
because it moves the execution context as well as data. Thus time taken
by a thread migration with MOBA islarger than latency of a remote invocation of ORBs, but the data transfer throughput is better with MOBA
than with the other systems.

6

Conclusion

A migration system for Java threads
has been implemented as a plug-in
to an existing JVM, and it supports
asynchronous migration of execution
context. Some problems pointed anddiscussed here were whether objects
reachable from the migrant should be
moved, how the types of values in thestack can be identiﬁed, compatibility with JIT compilers, and how resources tied to moving objects should
be handled.
As a further study, we are designing
a JIT compiler that can work wellwith thread migration. Hereby asynchronous and heterogeneous migration with execution of native code
will be got possible. And we havealready implemented a distributed
object system based on the JIT
compiler. So if it is integrated with
MOBA, selective migration of objects reachable from the migratory
thread and leaving the objects tied
to local resources will be possible.

References[1] A. Acharya, M. Ranganathan, and
J. Saltz. Sumatra: A language forresource-aware mobile programs.
In J. Vitek and C. Tschudin,
editors, Mobile Object Systems.
Springer Verlag Lecture Notes in
Computer Science, 1997.
[2] O. Agesen. GC points in a threaded
environment.
Technical Report
SMLI
TR-98-70, Sun Microsystems, Inc.,
Dec. 1998. http://www.sun.com/
research/jtech/pubs/.
[3] B. Dimitrov and V. Rego. Arachne:

A
portablethreads
system supporting migrant threads
on heterogeneous network farms.
IEEE Transaction on Parallel andDistributed Systems, 9(5):459–469,
May 1998.
[4] M. R. Eskicioğlu. Design issues ofprocess
migration facilities in distributed
system. IEEE Technical Comittee
on Operating Systems Newsletter,4(2):3–13, Winter 1989. Reprinted
in Scheduling and Load Balancing
in Parallel and Distributed Systems,
IEEE Computer Society Press.
[5] S.
Fünfrocken.
Transparent migration of javabased mobile agents. In Proc.
of 2nd Int’l Workshop on Mobile
Agents 98(MA’98), pages 26–37,
Sept. 1998.
[6] General Magic, Inc.
Odyssey
information.
http://www.genmagic.com/
technology/odyssey.html.
[7] S. Hirano. HORB: Distributed
executionof Java programs. In Proceedingsof World Wide Computing and Its
Applications, Mar. 1997.
[8] E. Jul, H. Levy, N. Hutchinson, and
A. Black. Fine-grained mobility
in the emerald system.
ACM
Transaction on Computer Systems,
6(1):109–133, Feb. 1988.

[11] ObjectSpace, Inc. Voyager.
http://www.objectspace.com/
products/Voyager/.
[12] M. Ranganathan, A. Acharya,
S. Sharma, and J. Saltz. Networkaware mobile programs.
In
Proceedings of USENIX’97, Jan.
1997.
[13] T.
Sekiguchi.
JavaGoManual, 1998. http://web.yl.is.s.utokyo.ac.jp/amo/JavaGo/doc/.
[14] T. Sekiguchi, H. Masuhara, andA. Yonezawa. A simple extensionofJava
languageforcontrollable transparent migrationand its portable implementation.
In To appear in a Springer
Lecture Notes in Computer Sciencefor International Conference on
Coordination
ModelsandLanguages(Coordination99), 1999.
[15] K. SHUDO. shuJIT—JIT compiler
forSun
JVM/x86.
http://www.shudo.net/jit/.
[16] Sun Microsystems, Inc. The JavaHotSpotT M
Performance Engine Architecture.
http://www.javasoft.com/
products/hotspot/
whitepaper.html.

[9] D.
Lange
and M. Oshima. Programming and
Deploying Java Mobile Agents with
Aglets. Addison Wesley Longman,
Inc., 1998.

[17] M. M. Theimer and B. Hayes.
Heterogeneous process migration
by recompilation. In Proc. IEEE
11th International Conference onDistributed Computing Systems,
pages 18–25, 1991. Reprinted in
Scheduling and Load Balancing in
Parallel and Distributed Systems,
IEEE Computer Society Press.

[10] T.
Lindholmand F. Yellin. The JavaT M Virtual
Machine Specification. Addison
Wesley, 1997.

[18] J. E. White. Telescript Technology:
The Foundation of the Electronic
Marketplace. General Magic, Inc.,1994.

[19] A. Wollrath, R. Riggs, andJ. Waldo. A Distributed Object
Model for the Java(tm) System. In
The Second Conference on Object–
Oriented Technology and Systems(COOTS) Proceedings, pages 219–
231, 1996.