Recovering UML Class Models from C++: A Detailed Explanation Abstract The software industry has widely accepted and often uses UML (Unified Modeling Language) [Fowler'00] tools in forward engineering, but these tools are used less frequently during software maintenance and evolution. This is due to a number of reasons; foremost among these is that the manual recovery and maintenance of UML models is time consuming and costly. As such, UML models become stale while the source code continues to evolve. Although many UML modeling tools allow us to reverse engineer UML models from source code, they often perform poorly at this task. A case study of reverse engineering tools [Kollman'02] finds that many of these tools, despite advances in the research literature, continue to focus on producing the core elements of UML (i.e., simple class diagrams), but often fail to adequately represent design abstractions. This is problematic when recovered software models fail to accurately represent the abstract program semantics required for high-level program comprehension. This problem can be exacerbated by the fact that end users are typically unaware of the internal processes for producing the UML models.. analysis system [Baxter'04] . It integrates the technologies used in reverse engineering such as static analysis, concept analysis and software clustering. More precisely, Figure 1 illustrates an architecture (or reference model) for a reverse engineering environment. This architecture is composed of layered analysis methods and models. Each method computes a model of the system that is consumed or used by higher level analyses (e.g., parsing yields an AST that can be used to compute control flow graphs). Moreover, there are a number of potential metrics (those shown are only a small subset of software metrics) that can be computed from the models at all layers of this architecture.. Much like the computation of metrics, mapping rules (or simply mappings) can be defined at any level of abstraction. Although these mappings can be used to produce any number of artifacts, here we specifically envision them being used to produce UML models. For example, simple mappings from C++ to UML produce UML classes from the C++ classes found in source code. However, these simple mappings -those most often implemented in common UML reverse engineering tools -are fairly naïve. They rarely embed more than a rudimentary knowledge of programming language semantics or libraries, nor do they allow users to embed their own specific domain knowledge into the reverse engineering processes. Moreover, with a limited set of analysis methods, these tools are often incapable of producing anything more than visually redocumented source code. Reverse engineered artifacts of this nature are often too detailed to be of any great use to the casual reader. The information they relate is easily attainable from the source code, or through a source code redocumentation tools such as Doxygen or JavaDoc.. The use of domain knowledge within these mappings provides a significantly broader range of functionality. Domain knowledge can includes programming style conventions (e.g., idioms, patterns, and even method naming), knowledge of reuse libraries, and even information from the problem domain. Additionally, these mappings can leverage other abstract source models (e.g., control flow, data flow, and call graphs) to assist in the mapping process. Table 1 describes the UML concepts [OMG'03] to be reverse engineered from source code. These concepts are representative of current deficiencies in reverse engineering tools as noted in [Kollman'02] and through our practical experience with a number of reverse engineering tools. Although the number of tasks required to fully reverse engineer C++ programs is significantly longer, we are interested in defining mappings for a small subset of those.. The columns in Table 1 are described as follows: the required analysis column describes the analysis method required to accomplish each task. This classification is based primarily on experience with C++ parsing technology and the implementation of different reverse engineering methods. Parsing implies that the results could be obtained through the abstract syntax tree (AST) of the program, semantic analysis means that the application must rely upon specific C++ semantics in order to perform the task, and static program analysis (or simply static analysis) indicates the need for more sophisticated analysis methods (e.g., data flow analysis). The domain knowledge column indicates whether or not domain knowledge will contribute to the completion of the task. Here, domain knowledge is taken to include knowledge of the problem domain and (primarily) the solution domain of an application and its implementation. For example, knowledge of programming idioms, design patterns, and code concepts (e.g., STL template concepts) are potentially useful when recovering the semantics source code. Tasks in which the application of domain knowledge plays a role are marked. The degree of ambiguity describes how consistently different tools perform the task. Low ambiguity implies that tools produce mostly consistent models, whereas high ambiguity tasks result in widely varying results. The degree of ambiguity was determined by considering the number alternative design-level semantics for a given C++ declaration. For example, identifying UML associations is highly ambiguous; this becomes evident when comparing the results of different reverse engineering application. The description column provides additional details for the reverse engineering task. Note that we have not included methods (member functions) in this table. Most reverse engineering tools typically have little difficulty identifying the methods of a class. Here we are interested in which methods are actually produced in the UML model. Most tools include constructors, destructors, and overloaded operations in the reverse engineered classes. However, we see these as implementation details (i.e., language integration features) that only serve to pollute the resultant UML model. We recommended allowing the user to decide the relevance of those methods to their comprehension requirements. In this paper we have discussed the inconsistency of reverse engineering tools due to the semantic gap between UML and C++ and the non-disclosure policy of those tools. In an effort to bridge the gap between the two languages and to provide a platform for common modeling problems, we have detailed a set of mappings from C++ to UML class models. These heuristic mappings are based primarily on easily accessible syntactic and semantic information in the program. These mappings are intended to recover design-level UML class models from source code, supporting program comprehension.. To validate the applicability and accuracy of these mappings, we have implemented them in our own reverse engineering application pilfer. When used to reverse engineer HippoDraw, it produced UML models comparable to those produced by other reverse engineering tools. However, the inclusion of domain knowledge allowed pilfer to produce models that reflected the abstract design rather than the simply recreating the implementation-level structures of the program. This is immediately obvious in a) the identification of abstract attributes and b) the presence of appropriate aggregation semantics in associations.. In the future we plan to expand our investigation of common ambiguities and inconsistencies between reverse engineering tools and define similar mappings for their resolution. Moreover, we plan to refine the architecture of pilfer to allow even more user customization, scriptability, and interoperability. Allowing users to develop and integrate more analysis technologies is a priority for creating a more complete and accurate reverse engineering environment. Although pilfer is already capable of producing standards-compliant XMI for model interchange, and dot files for visualization of UML diagrams and concept lattices, we envision a broader range of exported formats such that we can interoperate with a broad range of applications in the reverse engineering community. Efforts are being undertaken by the research community to define such formats [Holt'00, Malton'05] . Moreover, interoperability is being increasingly discussed as a requirement for reverse engineering tools as a whole (e.g., [Kraft'05] ). To support these broader goals of the community, we envision using pilfer to export, say, GXL models of call graphs, object relation diagrams and other such artifacts, allowing other applications to analyze or visualize the results. The prevailing method of integrating modeling and reverse engineering tools is to build reverse engineering parsers and analyses into existing UML modeling applications. Examples include Rational Rose, Together, Umbrello, Visual Paradigm, and ArgoUML. However, IDE's are beginning to realize the importance of providing a visual medium for source code and have begun to include UML modeling functionality. Both Microsoft's Visual Studio 2005 and Apple's XCode2 both support the ability to model classes in UML-like diagrams.. It is a matter of fact that most advances in reverse engineering research have little impact on industrial reverse engineering tools despite the fact that research technologies are addressing many of the shortcomings of industry applications. For example, numerous approaches have been described for reverse engineering associations and related information from Java. [Barowski'02] proposes a technique to extract dependency information from Java class (bytecode) files. The approach is used to detect inheritance, realization, and associations between classes. However, this definition of dependency is overly general and does not correspond to more specific relationships of UML. Moreover, the approach is not capable of expressing multiplicity or aggregation semantics of the recovered associations. [Guéhéneuc'04] provides a much stronger definition of associations at both the design and implementation level for both aggregate and composite associations. An algorithm for the detection of these associations is derived from a formal analysis of their properties and applied to a number of Java systems, performing quite well (96% recall and 75% precision). The approach described in [Jackson'99] applies heuristics to static and semantic analysis of Java class files. This work is very much like our own approach as it constructs object models (class models) using lightweight syntactic and semantic analysis. However, their implementation Womble, only operates on Java bytecode whereas pilfer, operates on C++ source code.. In [Gogolla'00] , UML associations, multiplicities and aggregation semantics are inferred by examining the source code. Additionally, this approach defines one of the few techniques for finding bi-directional associations. [Kollman'01] uses both static and dynamic analysis to accomplish many of the same goals. Yet another approach to identifying the multiplicity of associations is given in [Keschenau'04] . To our knowledge (possibly because of poor documentation), none of these approaches have been integrated into any of the prevalent reverse engineering tools.. Many of the analysis methods in the works listed above are not applicable to C++, especially those operating on bytecode. Unfortunately, related research on reverse engineering methods for C++ is significantly less. In [Tonella'01] , type information is reverse engineered from C++ by examining usage patterns of so-called weakly-typed containers. This approach can be used to identify which classes are used in containers if the container collects instances of abstract base classes such as the ubiquitous Object class. Although [Tonella'02, '03] focus more on recovering behavioral aspects of a program, they illustrate additional techniques for reverse engineering information from C++. In these approaches, an object flow graph is coupled with other forms of analysis to produce object and interaction diagrams, respectively. [Matzko'02] takes an approach similar to ours -the implementation of a C++ reverse engineering environment. However, this work focused on the modeling of C++ syntax in UML rather than attempting to recover the design abstractions involved. Moreover, little is said about the rules used to recover multiplicities or aggregation semantics.. Formal concept analysis is being applied to a number of different areas in software engineering many of which are surveyed in [Tilley'03] . In legacy systems, formal concept analysis has been applied to preprocessor configurations in order to better understand and help maintain complicated portability configurations [Krone'94, Snelting'98] . Concept analysis is frequently used to construct object oriented models from legacy or procedural code. Examples include [Canfora'99, Sahraoui'97, Siff'99, van Deursen'99] . It has also been used to analyze the coupling and cohesion between modules in [Lindig'97] .. Another popular application of concept analysis is to use it to reorganize or analyze class hierarchies in objectoriented software [Godin'93, Godin'98, Snelting'00, Snelting'98] . In [Tonella'99] , concept lattices are used to detect design patterns by analyzing inheritance relationships. Additionally, concept lattices are used as a visualization technique for information extracted from source code in [Cole'03] .. One innovative use of concept analysis has been in the area of program comprehension via feature location [Eisenbarth'01, '03] . In this research, concept analysis is used in conjunction with both dynamic and static analysis. Specifically, dynamic analysis is used to generate sets of related scenarios and subprograms. Concept analysis is used to identify features by relating which subprograms are invoked in each scenario. Static (call graph) analysis is then used to augment the information in the concept lattice, providing a detailed visualization the feature's implementation.. What distinguishes this work from other applications of concept analysis is primarily the scope to which the analysis is applied. These applications typically consider the entire scope of a program -sometimes generating an enormous amount of information that resists further analysis, much less visualization [Anquetil'00] . In our approach, we consider object/attribute on a class-by-class basis rather than the entire system as a whole. One application similar to ours is [Dekel'03] , in which concept lattices are constructed for individual Java classes to support program comprehension. Our approach varies in that the resulting lattices are used to determine additional qualities of a class rather than producing visualization material. The pilfer reverse engineering tool is currently implemented in the Python programming language. This was chosen for a number of reasons. First, it allows the application to be built and modified quickly, allowing developers to modify or experiment with the given mappings. pilfer leverages two key technologies to implement its reverse engineering capabilities: srcML 2 and the Open Modeling Framework (OMF) 3 . These technologies are explained in sections 4.1 and 4.2 respectively. In addition to the software used to implement the source code parsing and UML modeling, pilfer implements the mappings described above. Additional analysis components are also present in the implementation -specifically for the detection of attributes from C++ member variables.. The architecture and workflow for pilfer is shown in Figure 6 , which describes the interaction between the different components of the tool chain. Specifically, srcML is used to produce an XML representation of the source code. pilfer uses the srcML output to construct an abstract semantics graph (ASG), perform semantic analysis and execute the mappings over the resulting graph. Finally, UML model elements are constructed and serialized through the OMF. This section describes these components in detail.

O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


vrmlgen:

An

R

Package

for

Data

Visualization

on

the

Web

The

representation

and

inspection

of

complex

data

is

a

frequently

used

strategy

in

many

data

analysis

domains.

Existing

data

mining

software

often

lacks

functionality

that

would

enable

users

to

explore

data

interactively,

especially

if

one

wishes

to

make

dynamic

graphical

representations

directly

viewable

on

the

web.In

this

paper

we

present

vrmlgen,

a

software

package

for

the

statistical

programming

language

R

to

create

data

visualizations

in

web

formats

like

the

Virtual

Reality

Markup

Language

(VRML)

and

vrmlgen

can

be

used

to

generate

charts

and

bar

plots,

scatter

plots

with

density

estimation

contour

surfaces,

and

visualizations

of

height

maps,

object

models

and

parametric

functions.

For

greater

flexibility,

the

user

can

also

access

low-level

plotting

methods

through

a

unified

interface

and

freely

group

different

function

calls

together

to

create

new

higher-level

plotting

methods.

Additionally,

we

present

a

web

tool

allowing

users

to

visualize

data

online

and

test

some

of

vrmlgen's

features

without

the

need

to

install

any

software

on

their

computer.

visualizations

for

data

mining

provide

an

intuitive

means

to

identify

underlying

structures

and

patterns

in

a

data

set

which

would

otherwise

often

remain

undetected..

Software

tools

to

create

interactive

graphical

data

representations

in

can

be

used

effectively

both

to

explore

inherently

data

and

subspaces

and

projections

of

higher

dimensional

data.

The

latter

task

is

becoming

increasingly

important,

as

companies

and

research

institutions

are

striving

to

fully

exploit

their

vast

data

resources.

Moreover,

in

the

last

decade,

a

large

number

of

data

bases

containing

complex,

high-dimensional

data

have

been

published

for

general

access

on

the

web.

These

include,

for

example,

data

resources

for

the

life

sciences

(Edgar,

Domrachev,

and

Lash

Widera,

Garibaldi,

and

Krasnogor

World

Health

Organization

physical

sciences

(Zucker,

Kishore,

Sukumar,

and

Dragoset

CFADC

For

these

reasons,

a

great

variety

of

software

tools

has

been

made

available

in

the

past

to

generate

data

representations

with

interactive

means

to

explore

different

perspectives,

analyze

class

membership

and

data

density

using

color

schemes

and

contour

surfaces,

or

compare

and

interlink

different

data

sources

using

overlay-plots

and

hyperlinks.

The

widely

used

statistical

programming

language

R

(R

Development

Core

Team

for

example,

contains

several

software

packages

for

data

analysis

including

a

package

to

interconnect

R

with

OpenGL

(Adler

and

Murdoch

and

various

packages

for

multivariate

data

visualization

Tierney

Sarkar

Sarkar

Maechler

see

Section

for

a

more

detailed

discussion..

However,

though

users

can

choose

between

many

freely

available

offline-tools

to

inspect

data

on

their

own

computer,

it

is

still

a

difficult

task

to

make

visualizations

publicly

available

and

directly

viewable

on

the

web.

Currently

available

software

tools

or

packages

for

plotting

do

not

provide

features

for

a

web-based

interactive

exploration

of

the

data.

Apart

from

speed

limits

on

the

internet

for

loading

large

data

sets,

other

restrictions

arise

from

the

fact

that

different

web

browsers

do

not

support

the

same

graphics

file

formats

and

the

visualization

plugins

are

often

only

compatible

with

a

limited

number

of

browsers..

Here,

we

present

vrmlgen,

a

software

package

for

the

R

statistical

data

analysis

environment,

which

enables

users

to

generate

interactive

web-based

visualizations

for

input

data.

Charts,

graphs,

bar

plots

and

scatter

plots,

meshes

and

parametric

functions

are

visualized

in

two

common

web

formats,

VRML

(Virtual

Reality

Markup

Language)

and

Files

in

the

common

VRML

format

(Ames,

Nadeau,

and

Moreland

enable

an

interactive

virtual

navigation

in

complex

models

and

can

be

displayed

on

the

web

with

free

plugins

available

for

every

common

browser

platform.

The

Java-based

format

by

Kraus

which

is

supported

as

an

alternative

to

VRML,

provides

a

somewhat

reduced

user

interaction

functionality

and

is

based

on

the

simple

"painter's

algorithm"

to

hide

occluded

surfaces,

but

it

can

be

viewed

in

any

Java-enabled

browser

without

the

need

to

install

a

plugin.

vrmlgen

is

freely

available

subject

to

the

GNU

General

Public

Licence

(GPL

version

In

addition

to

the

R

package,

we

provide

a

simple

web

application

to

illustrate

some

of

vrmlgen's

main

features

and

enable

the

user

to

easily

generate

colored

and

labeled

visualizations

for

uploaded

data

without

the

need

to

install

any

software

(available

at

http:

Similarly,

the

R

package

is

employed

on

a

bioscientific

web

server

for

DNA

microarray

analysis,

ArrayMining

(Glaab,

Garibaldi,

and

Krasnogor

Infobiotics

Project

to

visualize

clustering

results

in

a

low-dimensional

representation..

The

structure

of

the

paper

is

as

follows:

We

will

present

vrmlgen's

features

and

code

examples

(Section

discuss

the

implementation

(Section

extensions

and

related

software

(Section

provide

installation

instructions

(Section

and

we

conclude

the

paper

with

a

summary

and

conclusion

in

Section

vrmlgen

and

the

R

software

environment

can

be

obtained

from

the

Infobiotics

project

repository

and

the

Comprehensive

R

Archive

Network

(CRAN)

at

VRML

files

can

be

inspected

either

with

offline

viewers,

e.g.,

BS

Contact

VRML

(Bitmanagement

vrmlgen

is

a

new

R

software

package

to

generate

visualizations

for

interactive

data

exploration

on

the

web.

It

provides

functions

for

the

creation

of

scatter

and

bar

plots,

and

visualizations

of

meshes,

parametric

functions

and

height

maps,

as

well

as

access

to

lowlevel

plotting

functions.

Calls

of

different

plotting

functions

can

be

grouped

together

to

generate

complex

scenes,

and

the

user

can

write

new

higher-level

plotting

methods

based

on

the

already

existing

functionality.

Additional

features

include

the

possibility

to

assign

hyperlinks

and

meta-labels

to

the

data

points

and

to

visualize

regions

of

high

data

density

by

means

of

contour

surfaces.

All

outputs

are

provided

in

two

common

web

formats,

the

VRML

and

format.

Apart

from

using

the

obtained

models

for

data

presentation

on

the

web,

these

files

can

also

serve

as

input

for

freeware

rendering

software

to

generate

high-quality

perspective

plots

for

scientific

publications..

For

demonstration

purposes,

vrmlgen

has

been

inter-linked

with

a

PHP

interface

to

enable

users

to

generate

visualizations

of

uploaded

data

matrices

on

the

web

without

the

need

to

install

any

software

Moreover,

the

R

package

is

employed

in

practice

on

a

bioinformatics

web

server

for

DNA-chip

analysis

(Glaab

et

al.

Infobiotics

Project

These

example

applications

show

that

vrmlgen

can

readily

be

used

within

web

applications

to

generate

visualizations

dynamically

in

response

to

user

activity

on

a

web

site.

vrmlgen

is

written

entirely

in

the

R

programming

language

and

can

be

installed

on

all

operating

systems

for

which

the

R

software

is

available.

The

variable

names

in

the

function

bodies

have

been

adapted

to

follow

common

R

naming

conventions

and

help

and

examples

for

all

functions

can

be

obtained

by

typing

?vrmlgen

at

the

R

command

prompt.

A

dedicated

website

with

installation

and

usage

instructions

as

well

as

example

output

pictures

is

available

at

Additionally,

we

have

implemented

a

web

tool

for

demonstration

purposes

(see

previous

section),

which

is

based

on

a

PHP-interface

running

on

an

Apache

web

server

and

enables

users

to

access

some

of

vrmlgen's

main

functions

without

the

need

for

a

local

software

installation..

The

R

package

was

build

following

the

recommendations

in

the

official

"Writing

R

Extensions"

tutorial

by

the

R

Development

Core

Team

and

has

been

published

in

the

Infobiotics

project

repository

as

well

as

in

the

CRAN

archive

(http:

//CRAN.R-Project.org/).

O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Dynamic consistency for stochastic optimal control problems Abstract. For a sequence of dynamic optimization problems, we aim at discussing a notion of consistency over time. This notion can be informally introduced as follows. At the very first time step t 0 , the decision maker formulates an optimization problem that yields optimal decision rules for all the forthcoming time step t 0 , t 1 , . . . , T ; at the next time step t 1 , he is able to formulate a new optimization problem starting at time t 1 that yields a new sequence of optimal decision rules. This process can be continued until final time T is reached. A family of optimization problems formulated in this way is said to be time consistent if the optimal strategies obtained when solving the original problem remain optimal for all subsequent problems. The notion of time consistency, well-known in the field of Economics, has been recently introduced in the context of risk measures, notably by Artzner et al. (2007) and studied in the Stochastic Programming framework by Shapiro (2009) and for Markov Decision Processes (MDP) by Ruszczynski (2009). We here link this notion with the concept of "state variable" in MDP, and show that a significant class of dynamic optimization problems are dynamically consistent, provided that an adequate state variable is chosen. Stochastic Optimal Control (SOC) is concerned with sequential decision-making under uncertainty. Consider a dynamical process that can be influenced by exogenous noises as well as decisions one has to make at every time step. The decision maker wants to optimize the behavior of the dynamical system (for instance, minimize a production cost) over a certain time horizon. As the system evolves, observations of the system are made; we here suppose that the decision maker is able to keep in memory all the past observations. Naturally, it is generally more profitable for him to adapt its decisions to the observations he makes of the system. He is hence looking for strategies rather than simple decisions. In other words, he is looking for applications that map every possible history of the observations to corresponding decisions. Because the number of time steps may be large, the representation of such an object is in general numerically intractable.. However, an amount of information lighter than the whole history of the system is often sufficient to make an optimal decision. In the seminal work of Bellman (1957) , the minimal information on the system that is necessary to make the optimal decision plays a crucial role; it is called the state variable (see Whittle, 1982 , for a more formal definition). Moreover, the Dynamic Programming (DP) principle provides a way to compute the optimal strategies when the state space dimension is not too large (see Bertsekas, 2000 , for a broad overview on DP). The aim of this paper is to establish a link between the concept of state variable and the notion of time consistency 1 . The notion of dynamic consistency is well-known in the field of economics (see Hammond, 1989) and has been introduced in the context of risk measures (see Artzner et al., 2007; Riedel, 2004; Detlefsen and Scandolo, 2005; Cheridito et al., 2006 , for definitions and properties of coherent and consistent dynamic risk measures). Dynamic consistency has then been studied in the stochastic programming framework by Shapiro (2009) and for Markov Decision Processes by Ruszczynski (2009) . In this paper, we rather use the (almost equivalent) definition of time consistency given by Ekeland and Lazrak (2006) , which is more intuitive and seems better suited in the framework of optimal control problems. In this context, the property of time consistency is loosely stated as follows. The decision maker formulates an optimization problem at time t 0 that yields a sequence of optimal decision rules for t 0 and for the following time steps t 1 , . . . , t N = T . Then, at the next time step t 1 , he formulates a new problem starting at t 1 that yields a new sequence of optimal decision rules from time steps t 1 to T . Suppose the process continues until time T is reached. The sequence of optimization problems is said to be dynamically consistent if the optimal strategies obtained when solving the original problem at time t 0 remain optimal for all subsequent problems. In other words, time consistency means that strategies obtained by solving the problem at the very first stage do not have to be questioned later on.. The notion of information here plays a crucial role. Indeed, we show in this paper that a sequence of problems may be consistent for some information structure while inconsistent for a different one. Consider for example a standard stochastic optimization problem solvable using DP. We will observe that the sequence of problems formulated after the original one at the later time steps are time consistent. Add now a probabilistic constraint involving the state at the final time T . We will show that such a constraint brings time inconsistency in the sense that optimal strategies based on the usual state variable have to be reconsidered at each time step. This is because, roughly speaking, a probabilistic constraint involves not only the state variable values but their probabilistic distributions. Hence the only knowledge of the usual state variable of the system is insufficient to write consistent problems at subsequent time steps. So, in addition to the usual technical difficulties regarding probabilistic constraints (mainly related to the non-convexity of the feasible set of strategies), an additional problem arises in the dynamic case. We will see that, in fact, this new matter comes from the information on which the optimal decision is based. Therefore, with a well-suited state variable, the sequence of problems regains dynamic consistency.. In §2, we carefully examine the notion of time consistency in the context of a deterministic optimal control problem. The main ideas of the paper are so explained and then extended, in §3, to a sequence of SOC problems. Next, in §4, we show that simply adding a probability constraint (or, equivalently in our context, an expectation constraint) to the problem makes time consistency fall apart, when using the original state variable. We then establish that time consistency can be recovered provided an adequate state variable is chosen. We conclude that, for a broad class of SOC problems, time consistency has to be considered with respect to the notion of a state variable and of DP. We informally introduced a notion of time consistency of a sequence of decisionmaking problems, which basically requires that plans that are made from the very first time remain optimal if one rewrites optimization problems at subsequent time steps. We show that, for several classes of optimal control problems, this concept is not new and can be directly linked with the notion of state variable, which is the minimal information one must use to be able to take the optimal decision.. We show that, in general, feedback laws have to depend on the probability law of the usual state variable for Stochastic Optimal Control problems to be time consistent. This is necessary, for example, when the model contains expectation or chance constraints.. Future works will focus on three main directions. The first concern will be to better formalize the state notion in the vein of the works by Witsenhausen (1971 Witsenhausen ( , 1973 and Whittle (1982) . The second will be to establish the link with the literature concerning risk measures, in particular the work by Ruszczynski (2009) . Finally, the last DP equations we introduced are in general intractable. In a forthcoming paper, we will provide a way to get back to a finite-dimensional information variable, which makes a resolution by DP tractable.

B I O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Visions and open challenges for a knowledge-based culturomics The concept of culturomics was born out of the availability of massive amounts of textual data and the interest to make sense of cultural and language phenomena over time. Thus far however, culturomics has only made use of, and shown the great potential of, statistical methods. In this paper, we present a vision for a knowledge-based culturomics that complements traditional culturomics. We discuss the possibilities and challenges of combining knowledgebased methods with statistical methods and address major challenges that arise due to the nature of the data; diversity of sources, changes in language over time as well as temporal dynamics of information in general. We address all layers needed for knowledge-based culturomics, from natural language processing and relations to summaries and opinions. The need to understand and study our culture drives us to read books, explore Web sites and query search engines or encyclopedias for information and answers. Today, with the huge increase of historical documents made available we have a unique opportunity to learn about the past, from the past itself. Using the collections of projects like Project Gutenberg [67] or Google books [25] , we can directly access the historical source rather than read modern interpretations. Access is offered online and often minimal effort is necessary for searching and browsing. The increasing availability of digital documents, spread over a long timespan, opens up the possibilities to move beyond the study of individual resources. To study our history, culture and language, we can now computationally analyze the entire set of available documents to reveal information that was previously impossible to access.. The aim of the emerging field Culturomics, introduced by Aiden and Michel [3] and Michel et al. [53] , is to study human behaviors and cultural trends by analyzing massive amounts of textual data that nowadays are available in digital format. By taking a purely statistical view, Michel et al. [53] unveiled information in Google books that would not have been found without the analysis of this large text corpus. Already by studying word frequencies interesting linguistic and cultural phenomena can be found.. One example of a linguistic phenomenon is the size of the English lexicon. The numbers of unique common words for the years 1900, 1950 and 2000 were compared and the authors found that by year 2000, the size of the lexicon had increased significantly (but see [46] ). 1 This means that by the year 2000, more unique words are used in written text than ever before. As an example of a cultural phenomenon the authors studied fame, approximated by the frequency of a person's name. The 50 most famous people born each year between 1800 and 1950 were studied based on several criteria, among others the age of peak celebrity and the half-life of decline. The authors found that, over time, people become famous earlier in their lives and are more famous than ever but are being forgotten faster.. The examples given above showcase the potential of culturomics and could not have been found by studying individual resources or by exclusively using manual analysis. However, despite the large potential of a purely statistical view, culturomics would gain from complementing with deeper, knowledge-based approaches as well as integrate preexisting knowledge. In this paper, we introduce a knowledgebased culturomics that complements the purely statistical method of classic culturomics with NLP and recent information extraction techniques. We will present our vision and current progress toward a knowledge-based culturomics and discuss open challenges. The notion of culturomics has turned out to be useful and interesting, not only from a research perspective but also for raising interest in historical data. However, without links to modern data it is less likely that culturomics will gain much interest from the general public. By moving toward knowledge-based culturomics, many more possibilities will be opened.. By offering support for integrating textual sources with modern and historical language, we can better bring information from historical resources (and the resources themselves) to users. This also allows researchers to better track information over long periods of time, without language change and variation getting in the way.. To achieve the goals set in this paper, particular focus must be given to natural language processing techniques and resources for entity, event and role extraction that constitute the basis of knowledge-based culturomics. Using this firstorder information, knowledge-based culturomics opens up a large toolbox for researchers and the general public alike. Opinion mining and temporal semantic summaries allow for quick overviews and easy understanding of large amounts of textual resources over long periods of time. This analysis allows us to not only track information over time, but also clearly understand the change itself. As an example, we can find changes in opinion toward an entity over time as well as indications of the reason for the change, e.g., a specific event.. Finally, by linking extracted, second-order information to the primary resources, knowledge-based culturomics can offer credibility and traceability to this type of digital humanities research. Users can be offered a semantic interpretation of the important aspects of their queries as well as key resources that can serve as the starting point for further research.. While the benefits of knowledge-based culturomics are great, the challenges that lie ahead are equally large. Natural language processing is inherently hard as it is based on one of our most complex systems. However, tackling natural language at different periods in time, taking into consideration changes to the language, significantly increases the difficulty of the task. The increasing difficulty that comes with longer timespans applies to most other technologies that have thus far mostly been applied to data from short time periods. The strength of knowledge-based culturomics is the interconnected manner with which these challenges will be tackled, where the output of one technology can feed into another to improve the results. In culturomics, language change is a great obstacle for most other tasks like aligning and tracking opinions and topics or creating summaries.. Language change is interesting on its own; are there periods in which we change our language more than others? Is language change correlated to major events or cultural changes, e.g., introduction and wide spread use of Web log brought with it a whole set of new terms like blog, blogging, blogger. Are there other such events? Which type of change is more likely to be stable and which type of change is ephemeral and how have these classes changed over time? Did the introduction of user-generated content increase the amount of ephemeral changes?. The technical challenges lie in the lack of datasets and methods for automatic testing and evaluation over long timespans. Typically, tasks like word sense discrimination or named entity evolution have been developed and evaluated on short, modern timespans. While certain tasks are simpler to evaluate (e.g., named entity evolution) tasks like word sense discrimination or evolution are significantly harder, in particular when long timespans are taken into account and manual evaluation becomes infeasible.. Opinion mining can help to answer many culturomics research questions by providing insight into the opinions and values expressed in a document collection. The unique challenges of culturomics, such as the diversity of document sources and topics, as well as the longer time periods, have however not been tackled in previous work. As the techniques for opinion mining mature, these challenges will need to be addressed, especially as they also increase the robustness of opinion mining for more general applications.. A particular area where knowledge-based culturomics can help to drive further research is the detection of slow changes in cultural opinions and values. These manifest themselves through opinions expressed toward groups of related entities from different time periods that form the topic of the opinion change topic. Tools like automatic relation extraction (see Sect. 2.2) and Named Entity Evolution Recognition (see Sect. 3) can help us find the relevant entities, so that we can find all relevant opinions and analyze their dynamics. Furthermore, key resource finding (see Sect. 6) can help us corroborate the detected changes with relevant source documents.. Temporal semantic summarization can help users deal with information overload, and follow developments within topics of interest over lengthier periods of time.. As the method of scoring information in the documents will rely on the success of extracting information, a great challenge lies in combining all these techniques in the best possible way. The aim is to develop a system that works on big amounts of input text, an aspect that can be seen both as a challenge and a source of strength. Algorithms need to be able to cope with the volume, variety and velocity of large amounts of data as well as be robust against noise (e.g., OCR errors or spelling variations in text).. An important aspect of a good summary in a culturomics context is completeness as well as order. A summary should contain at least the major events (e.g., marriage, mergers, battles) and the strongest relations (e.g., people, places and titles), all sorted in time. For temporal ordering, an important aspect becomes to sort information from documents written about an event during the event compared to information found in documents written after a significant time has passed. To sort the latter type of information, publication time cannot be used, instead time references must be extracted from the text itself, introducing an additional difficulty.. Finally, a proper summary should contain not only facts but also different viewpoints. Relying on the output of the opinion mining step, methods should be found to bring the opinions into the summary, also these in a temporal order. E.g., what were the different viewpoints of the Vietnam war during the war compared to the viewpoints after the war, and in particular, what are the differences.. An interesting addition to both summaries (and opinions) is the differences between automatically created summaries using information available during an event compared to information available post-event. What has been learnt afterwards and what has been forgotten? Which events or relations were deemed important in hindsight that were not considered important during the course of an event and vice versa?. An important aspect of key resource finding is to step away from exact matching of words and sentences and allow for semantic similarity that captures, e.g., similarity between the words guitar and music because the guitar is an instrument used for creating music. For modern data, resources like WordNet [56] or DBpedia [12] , their hierarchies or explicit relations can be used. For historical data where such resources may not be adequate, semantic similarity must be found using a combination of contemporary resources and unsupervised information extraction techniques. Because a large focus is given to historical resources in culturomics, finding semantic similarity measures and techniques for key resource finding is a challenge of great importance and future work is to investigate the utility of continuous word representations with temporal variance.. As culturomics moves toward deeper analysis of large-scale corpora, the demand on processing capability moves beyond what any single computer can deliver regardless of its mem-ory size or processor speed. This is a particularly important point as we move toward knowledge-based culturomics where many processes need more than one pass through the data and are computationally heavy. With the big data era, a new set of frameworks and tools have emerged that simplify the deployment of a distributed computing framework. To handle these challenges, focus must be given to largescale distributed processing, for example, on frameworks like Hadoop [86] .. Much of the discussion in this paper has regarded historical collections and the type of noise that must be handled with regard to, e.g., OCR errors and language change. However, it is important to keep in mind that similar problems arise when dealing with modern data, in particular from usergenerated sources. A great variety in the form of slang, abbreviations and variations in spelling, word usage and grammar are present that can affect, e.g., NLP and semantic role labeling. To succeed with knowledge-based culturomics, attention must be paid to handle all these kinds of variation and provide methods that are robust against noise regardless of its origin.. One great advantage of knowledge-based culturomics is the possibility to iteratively apply different technologies and allow for automatic or semi-automatic improvements. As an example, while clustering word senses, we find that spelling and OCR errors often end up in the same clusters. Hence, using word sense clustering we can automatically determine which words to correct by, e.g., correcting words with low Levenshtein distance [45] to the word with the most frequent spelling. These corrections can be applied to the entire dataset and then clustering can take place again leading to higher quality clusters and possibly more or larger clusters. A concrete example is the following cluster derived from the Swedish Kubhist diachronic news corpus [80] , for the year 1908: (carbolinenm, kalk, cement, carbolineam, eldfast, carbolineum). Three different spellings of carbolineum are present in the same cluster, two of them OCR errors which now can be corrected.. The strength of classical culturomics lies in its simplicity and low computational requirements. Because of the amount of data involved, many errors become statistically irrelevant and interesting patterns appear in the combination of (a) amount of data and (b) large timespans. However, there are many questions that cannot be answered using traditional culturomics; following concepts or entities over time that have changed their names or lexical representations, or distinguishing between different entities with the same name. Adding sentiment analysis, detecting changes in sentiment and the events that caused the changes is one example of moving beyond traditional culturomics.. Still, a deeper understanding of culturomics data requires human analysis. With knowledge-based culturomics the relevant resources can be found to help users focus their attention. Research questions like "what were the most relevant aspects of Marie Antoinette's life in terms of people, places, relations, what was the general sentiment of her and how and why did it change" can be answered. For each of these questions, documents can be provided for deeper analysis, going far beyond information retrieval with term matching techniques.. Current methods for NLP and statistical culturomics typically have the implicit assumption that the information expressed in documents is correct and can be taken at face value. For example, in opinion mining it is often assumed that the opinion expressed is the actual opinion of the speaker. First steps have been done to detected irony and sarcasm, where the implicit assumption obviously does not hold. However, this does not cover cases where authors misrepresent their opinions or factual knowledge for personal, political or social reason, e.g., to gain a financial advantage when talking about company stocks they own or to improve their standing in their peer group. Knowledge-based culturomics can be a first step to challenge the implicit assumption in two different ways: first, the richer model of the connections between entities, authors and documents makes it possible to aggregate first-order information in ways that are more representative than simple statistical summation. Furthermore, the findings of knowledge-based culturomics methods are linked back to source documents using key resource finding. This helps researchers validate the results instead of having to blindly trust the algorithms.. An interesting development in natural language processing, that we predict will have a strong impact on culturomics in the future, is the movement toward continuous vector space representation of words. These models embed words in a vector space that reveals semantic and syntactic similarities and can be used to lift the concrete text to a higher level of abstraction. This property was leveraged by Kågebäck et al. [36] to enhance the state-of-art extractive summarization algorithm introduced by Lin and Bilmes [47] , using word embeddings to compare the information content in sentences. Further, these representations have been shown to exhibit interesting compositional properties, e.g., senses such as plurality and gender are captured as a linear translations in vector space, which can be exploited to draw conclusions not directly apparent from the text. An example of this is analogy testing, Mikolov et al. [55] , where the question A relates to B as C relates to? Is answered using simple vector arithmetics (v B − v A ) + v C ≈ v D where v i denotes a vector representation of word i and v D the vector representation of the sought answer. In culturomics word embeddings could be used for a variety of tasks, significantly extending the scope and impact of the project developed by Aiden and Michel [3] via the use of much richer representations in place of skipgrams. As a first example, we could directly compare words from different time periods by computing word embeddings on time slices of history and projecting them into the same space, revealing trajectories for each word that plot the evolution of each corresponding word in a semantic space. Another example is to use the compositionality property, and compute how the vector space captures the popularity of a countries leader and subsequently track the evolution of a leaders corresponding word embedding for signs of dropping popularity, providing an early warning for civil unrest.. The aggregated data show that results of knowledge-based culturomics can be interesting for the general public and can help raise interest for historical resources. Imagine connecting modern newspaper articles with historical archives and for interesting current events being able to present the historical view. As an example, in an article on the topic the Thailand Army declares martial law after unrest, also historical aspects leading up to the event can be presented to users in condensed formats. Important people, places, events and different sentiments in summarized formats with links to descriptive original articles are all important to complement the current information and provide a more complete view. Opinion extraction (or "opinion mining", "sentiment analysis", or "subjectivity analysis") is a wide and diverse field of research [50, 65] . Automatic retrieval of opinionated pieces of text may be carried out on a number of different levels of granularity. On the coarsest level, documents are categorized as opinionated or factual [90] . This may for instance be used to distinguish editorials from news [93] . Classification of sentences is also widely studied; these classifiers have been based on linguistic cues [87] or bag-of-word representations [64] . While most work use supervised machine learning, there are also unsupervised approaches [41, 48] .. In contrast to the early work, recent years have seen a shift toward more detailed and fine-grained problem formulations where the task is not only to find the text expressing the opinion, but also analyzing it: who holds the opinion (the holder) and toward what is it directed (the target); is it positive or negative (polarity); what is its intensity [16, 35, 39, 89] . The increasing complexity of representation leads us from retrieval and categorization deep into natural language processing territory; the methods employed here have been inspired by information extraction and semantic role labeling, combinatorial optimization and structured machine learning. For such tasks, deeper representations of linguistic structure have seen more use than in the coarse-grained case [27, 35, 39, 73, 76] . An extensive survey of opinion mining is provided by Pang and Lee [65] and more current developments are summarized by Tsytsarau and Palpanas [82] .. Previous work can roughly be divided into three different categories: extractive methods, abstractive methods, and orthogonally, methods that use information extraction.. In extractive summarization every sentence in the input documents is considered as a candidate. The task is to choose the most representative sentences, according to some metrics, to include in the summary. A common approach is to represent each sentence as a weighted vector of TF*IDF terms, and compute the cosine similarity between all pairs. Some extractive approaches start by deriving representations of the topics, whereafter the sentences are scored based on impor- Fig. 4 Illustration of (Extractive) Automatic Summarization tance. In other methods, the problem is seen as a trade-off between diversity and similarity. See [47, 54, 68] for more details.. In abstractive summarization, summaries are created by generating natural language sentences. All documents are parsed to build up knowledge in some representation and this knowledge is used to generate new sentences that summarize the documents. Ganesan et al. [23] , Leskovec et al. [44] , Rusu et al. [74] extracted subject, verb, object triplets from the documents to create a graph based on the semantic relations found. They use different techniques to extract important parts of this graph and generate the summary based on each part. While abstractive summarization has great potential, the inherent difficulties with changing language and large amount of noise in the collections used for culturomics leave abstractive methods for future work.. Recently, there has been some focus on using information extraction to improve summarization. Filatova and Hatzivassiloglou [21] presented a method that boosts event extraction coverage by relaxing the requirements for an event. Any pair of entities that appear in the same sentence and in combination with a connector word is considered an event. The connector word can be a verb or an action noun, as specified by WordNet. The sentences are scored using a greedy optimization algorithm and the results improve over an extractive baseline summarizer that did not account for redundancy.. Hachey [29] presented General Relation Extraction. To classify two entities as participating in a relation, they require the entities to be separated by no more than two words or by only one edge in a dependency parse tree. Connector words are derived from a model of relation types based on latent Dirichlet allocation, avoiding dependency on domainspecific resources like WordNet. The results were similar to those of Filatova and Hatzivassiloglou.. Ji et al. [34] used information extraction to perform relevance estimation and redundancy removal. This was done by combining the scores computed based on IE with scores based on coverage of bi-grams from the extractive summarizer used [24] . Combining these in the right way helped to create summaries that improved over the baselines.. All the above methods show an increased performance over an extractive summarization baseline. However, none perform any analysis on temporal information. Temporal summarization has been targeted in the past with a recent upswing [4, 11, 92] . Allan et al. [4] choose one sentence from each event within a news topic. Yan et al. [92] creates individual but correlated summaries on each date from a timestamped collection of Web documents. Binh Tran [11] first ranks and chooses the top time points and then chooses the top sentences for each of the chosen time point. All works make use of news articles where typically one document describes one event and timestamps are exact and narrow. Automatic event detection or relation extraction was not considered. To achieve high-quality temporal summaries on historical texts with long timespans, temporal summarization techniques must be combined with information extraction.. Key resource finding is an NLP task that falls under the category of information retrieval (IR). Traditional IR systems score the relevance of documents using keywordbased matching with respect to a given free text query. This approach has the advantage of being tractable for large-scale systems, but does not use higher level information in the text. Using semantic role labeling (SRL), it is possible to construct entity-relation graphs, that subsequently may be used to improve the precision of the search results.. Lin et al. [49] introduced an event-based information retrieval approach that finds relevant documents using entities and events extracted from the corpus. The query is created by the user in a structured form, but is limited to a single predicate. This approach could be generalized to the problem of finding a document describing a well-defined relation, as defined in Sect. 6.1.. A more general approach is employed by Kawahara et al. [38] , where a predicate-argument graph is used to represent both the query and the documents. Document selection is done using binary operators (i.e., AND OR) on the predicateargument structures as defined in the query. However, the method leaves SLR-based ranking as future work and does not provide a way to determine how many (or which) documents that are needed to cover the complete query, where the latter is necessary to answer the general question posed in Sect. 6.1.

O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Plug-in scheduler design for a distributed grid environment This paper presents the approach chosen within the DIET (Distributed Interactive Engineering Toolbox) project a Grid-RPC environment to allow a resource broker to be tuned for specific application classes. Our design allows the use of generic or application dependent performance measures in a simple and seamless way. The GridRPC approach [13] where remote servers execute computation requests on behalf of clients connected through the Internet is one of the most flexible and efficient solutions for porting large scale applications over the grid. In this model resource brokers (also called agents) are responsible for identifying the most appropriate servers for a given set of requests.. One of the main problems is, of course, to evaluate the costs associated with remote execution of these requests. Depending on the target application, several measurements have to be taken into account. Execution time is typically the measure with which users are most concerned, but others factors, such as available memory, disk, machine load, batch queue length may also be significant. Moreover, particular applications may have specific features and behaviors, which should also be considered. However, these various performance measures can only be taken into account if the middleware allows tuning of its internal scheduling software by the applications it services. * Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. This paper presents the approach chosen within the DIET (Distributed Interactive Engineering Toolbox) project to allow a resource broker to be tuned for specific application classes. Our design allows the use of generic or application dependent performance measures in a simple and seamless way. The remainder of the paper organized as follows: Section 3 presents the architecture of the DIET middleware framework. Section 4 describes the plug-in scheduler feature and Section 5 describes the CORI collector which allows the management of different performance measures. Finally, Section 6 presents some early experiments of this new feature of our GridRPC framework before a conclusion. In this paper, we described the design of a plug-in scheduler in a grid environment, and reported on an implementation of this design in the DIET toolkit. To provide underlying resource performance data needed for plug-in schedulers, we designed a tool that facilitates the management of different performance measures and different kinds of resource collector tools. As a proof of concept, we developed a basic resource information collector that enabled very basic performance data to be incorporated into DIET's scheduling logic. We then conducted two experiments that illustrated the potential benefits of utilizing such information in realistic distributed computing scenarios. As future work, we plan to use the plugin scheduler to design highly-tuned application-specific schedulers for several real applications that we have begun to study. We also intend to incorporate new collectors that seem promising in terms of providing resource performance data that may be useful in the DIET scheduling process. Several middleware frameworks follow the GridRPC API from the GGF like Ninf [9] or GridSolve [18] , but none of them expose interfaces that allow their scheduling internals to be tuned for specific application classes.. APST [4] allows some modifications of the internals of the scheduling phase, mainly to be able to choose the scheduling heuristic. Several heuristics can be used like Max-min, Min-min, or Xsufferage.. Scheduling heuristics for different application classes have been designed within the AppLeS [5] and GrADS [1] projects. GrADS is built upon three components [6] . A Program Preparation System handles application development, composition, and compilation, a Program Execution System provides on-line resource discovery, binding, application performance monitoring, and rescheduling. Finally, a binder performs a resource-specific compilation of the intermediate representation code before it is launched on the available resources. One interesting feature of GrADS is that it provides application specific performance models. Depending of the target application, these models can be built upon an analytic evaluation by experts joined with empirical models obtained by real world experiments. For some applications, simulation is used instead to guess their behavior on some specific architectures. These models are then fed into the schedulers to find the most appropriate resources used to run the application.. Within the Condor project [15] , the ClassAds language was developed [11] . This language allows to specify resource query requests with attributes built upon lists of constants, arbitrary collections of constants and variables combined with arithmetic and logic operators. The result is a multicriteria decision problem. The approach presented in our paper allows a scheduling framework to offer useful informations to a metascheduler like Condor.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O


Learning and Control Model of the Arm for Loading We propose a learning and control model of the arm for a loading task in which an object is loaded onto one hand with the other hand, in the sagittal plane. Postural control during object interactions provides important points to motor control theories in terms of how humans handle dynamics changes and use the information of prediction and sensory feedback. For the learning and control model, we coupled a feedback-errorlearning scheme with an Actor-Critic method used as a feedback controller. To overcome sensory delays, a feedforward dynamics model (FDM) was used in the sensory feedback path. We tested the proposed model in simulation using a two-joint arm with six muscles, each with time delays in muscle force generation. By applying the proposed model to the loading task, we showed that motor commands started increasing, before an object was loaded on, to stabilize arm posture. We also found that the FDM contributes to the stabilization by predicting how the hand changes based on contexts of the object and efferent signals. For comparison with other computational models, we present the simulation results of a minimumvariance model. Many studies have been done to find the mechanism that humans use to generate arm movements. Arm movements can be classified into two groups, postural control and reaching movements. Among the arm movements, mainly reaching movements have been studied, and the associated profiles, roughly straight hand paths with bell-shaped velocity, are well reconstructed by the assumption that the central nervous system (CNS) optimizes cost functions related to trajectory or muscle activations [1] - [3] . In the optimization models, predictions are mostly compared with horizontal reaching movements, which do not account for the influence of gravity. On the other hand, postural control has been studied relatively less, perhaps because postural control is considered comparatively easy or less important. However, it has its own complexities, especially in environments under sudden dynamics changes in the sagittal plane.. In such an environment, stable postural control will be difficult unless control problems (e.g. predictions of environmental changes, how to use visual and somatosensory information, delays in the sensory path) are correctly solved. Moreover, since there are infinitely numerous combinations of muscle activities for maintaining a position, postural control is an ill-posed problem. This is an important issue of motor control to be considered. In addition, arm movements in real life are affected by gravity and postural control occurs in many daily-life interactions, such as being handed a glass of water. Therefore, postural control in the sagittal plane is a valuable topic to study for generalizing motor control theories of the arm. In a psychophysical experiment where a subject performs a loading task, loading an object onto one hand with the other hand, in the sagittal plane, it is confirmed that muscle forces are gradually increasing before the object contacts the hand [4] . This result indicates that muscle forces are adjusted in a feedforward way. It is, however, not clear how humans can learn and control motor commands in a feedforward way without prior dynamics knowledge, nor what the motor control model consists of.. One viable controller for realizing the feedforward way in biological systems is to use an inverse model [5] . The inverse model inverts the causal flow of the biological system, which generates from a given state of the arm motor commands that produced that state. Is solely an inverse model enough as a controller of the arm? Humans are able to respond in environments with sudden dynamics changes, such as a loading task or catching a ball, despite time delays of about 50-100 ms in the somatosensory loop when transferring sensory signals from sensory receptors to effector organs (e.g. arms and legs). For this to be possible, we believe a motor control model needs a module to compensate for possible movement errors due to time delays. Errors would not be generated if the inverse model outputted accurate motor commands for all environmental changes. It is, however, quite unlikely to have accurate inverse models for all cases in our daily lives. Therefore, the motor control model needs a feedback controller. In addition, a feedforward dynamics model (FDM) would be used in the feedback loop for state predictions and acquiring context information from the environment [6] . The usage of the FDM in the brain when interacting with objects is represented by Grip-force-load-force Copyright c 2009 The Institute of Electronics, Information and Communication Engineers coupling [7] , [8] . It is also thought that the FDM minimizes the effects of sensory delays [9] , [10] .. In this study, we proposed a motor control model for loading based on the aforementioned principles. In the proposed model, an inverse statics model (ISM) is used as the inverse model, which handles the static component of the inverse dynamics. Therefore, when a desired state (e.g. joint angles) is given, the ISM outputs motor commands that make the arm converge to the corresponding equilibrium position. A feedback-error-learning scheme [11] is applied to train the ISM. Although Direct Inverse Modeling [12] and Forward and Inverse Modeling [9] can be used to train the inverse model, feedback-error-learning has more advantages because it acquires the inverse model while controlling biological systems that have redundancy. Alternatively, the Actor-Critic method which is one of the major frameworks for temporal difference (TD) learning in the Reinforcement Learning [13] is adapted to train the feedback controller in feedback-error-learning. Since it is difficult to imagine that we are born with a trained feedback controller, learning should be performed in a trial-error manner to acquire it [14] , [15] . Finally, a FDM is used in the feedback path to overcome sensory delays. The validity of the proposed framework was already shown in the previous study, [14] for reaching movements.. The objective of this study was to determine how humans control the arm in an environment where sudden dynamics changes occur (e.g. loading) and how to use the information of prediction and sensory feedback. Calling the proposed model a learning and control model, we use the term "learning" to convey that the proposed model can acquire control laws without prior dynamics knowledge. In the simulation of the proposed model, a loading task was performed. Hand trajectories, motor commands and hand force were represented using a two-joint arm with six muscles that have delays in signal transfer. Results showed that motor commands gradually increased, before the object made contact with the hand to stabilize posture based on predicted information from the FDM. The simulation data showed tendencies similar to the psychophysical experimental data in the view of hand trajectories and hand force. We introduced a learning and control model of the arm for a loading task in the sagittal plane. An Actor-Critic method is adapted in a feedback-error-learning scheme as a feedback controller. FDM is used in a feedback path to predict state and overcome sensory delays. Simply, the loading task can be performed if the stiffness of the muscles is kept high during the task. However, it is not a control policy that humans take. Humans gradually increase the motor commands before the environment changes to stabilize the hand trajectory because of sensory delays [4] . With this model, it was possible to perform the loading task in a human-like way, and tendencies of hand trajectories and hand force similar to the psychophysical experiment were observed. In addition, we showed that stable control in the loading task was possible due to the usage of the FDM.. The proposed framework used the feedback-errorlearning scheme proposed as a computational model of cerebellar motor learning [11] , [27] and the Actor-Critic method proposed as a model of learning in the basal ganglia [28] , [29] . However, we are not sure that the cerebellum and basal ganglia are directly related to performing the loading task, although those regions are possibly used to train the inverse model and the feedback controller, which are considered necessary for the loading task. Psychophysical experiments that measure brain activity (e.g. FMRI) during the loading task would help to identify the brain circuitry and understand the control mechanism.. In our previous study, we showed that the proposed framework can be applied to reaching movements in the sagittal plane and can successfully predict its profiles [14] . Through the simulation of the loading task, it is shown that a unified framework is applicable not only to reaching movements but also to postural control. We believe our findings will help to generalize motor control theories of the arm. in the Musculoskeletal System. In the simulation, the dynamics equations of the two-link arm in Here, τ j , θ j ,θ j andθ j indicate joint torque, angle, velocity and acceleration of the arm, respectively. Subscripts ( j = S , E) indicate the shoulder and elbow. m i , l i , l gi and I i are the weight of the arm, the length of each link, the length from the joint to the mass center of each link, and the moment of inertia around each joint. These parameters are defined as shown in Table A· 1. Besides, muscle parameters in the musculoskeletal system are defined as shown in Table A· 2 and Table A· 3 . k 1 and b 1 are the elastic and viscosity coefficients, respectively. k 0 and b 0 are the intrinsic elasticity and viscosity. l 0 is the intrinsic rest length. l 1 is the muscle length coefficient.. When an object is loaded on the hand, the joint torques for the object (τ 1), is required more when holding the object than when the object is not loaded to maintain the same posture. The FDM is simulated using the dynamics equations. The process is as follows. Firstly, the FDM judges from the context time, t contact − mΔt whether the object should be considered in the control machanism (i.e. when the time t passed over the context time, the object torque of Eq. (16) is considered. Otherwise, the object torque is set to 0). Secondly, torque in the arm at time t are calculated from the motor commands,ũ sum i using Eq. (14) and Eq. (15) . Then, the state of the arm is updated with the Runge-Kutta method and the time changes from t to t + Δt. This calculation is repeated until the time becomes t+nΔt. During the repeated calculations, the motor commandũ sum i is fixed on the value at time t.

O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O B O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O