Facilitating Benchmarks for 

Robotic Manipulation

Berk Calli

Robotics Engineering Department

Computer Science Department

Worcester Polytechnic Institute
Worcester Polytechnic Institute

Outline

• Efforts to enable benchmarking in robotic manipulation

YCB Project

is launched

IROS

Workshop

ICRA

Workshop

ICRA

Workshop

RA-Letters

Special Issue

Announcement

2014 2015 2017 2019 (Feb) 2019 (May) 2020 (Jan)

Special Issue 

Publication Feb

2 Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

Outline

• Efforts to enable benchmarking in robotic manipulation

YCB Project

is launched

2014

3 Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

[1] Andreas ten Pas, Robert Platt, Using Geometry to Detect Grasps in 3D Point Clouds, 2015

[2] Ravi Balasubramanian, Ling Xu, Peter D. Brook, Joshua R. Smith, Yoky Matsuoka, Human-Guided Grasp Measures Improve 

Grasp Robustness on Physical Robot, ICRA, 2010

[3] David Fischinger and Markus Vincze , Empty the Basket - A Shape Based Learning Approach for Grasping Piles of Unknown 

Objects, IROS 2012.

[4] Ashutosh Saxena Justin Driemeyer Andrew Y. Ng Robotic Grasping of Novel Objects using Vision, IJRR, 2008.

[1] [2] 

[3] [4] 

4
Worcester Polytechnic Institute

Facilitating Benchmarking 

in Robotic Manipulation

5

Aaron Dollar Berk Calli Sidd Srinivasa Aaron Walsman Pieter Abbeel Arjun Singh

/

YCB Project

is launched

2014

Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

Facilitating Benchmarking 

in Robotic Manipulation

6

/

Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

Benchmarking Tools

7 Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

YCB Object and Model Set

8

• Presented the set in the ICRA workshop

• Distributed it for free to the attendants

• Started online distribution (to what it costs)

• ~200 sets have been distributed so far

YCB Project

is launched

ICRA

Workshop

2014 2015

Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

YCB Object and Model Set

9

• The set was widely utilized by the research community

• However,

─ Experimental procedures are not standardized

─ Quantification metrics are not standardized

YCB Project

is launched

ICRA

Workshop

2014 2015

Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

Outline

• Efforts to enable benchmarking in robotic manipulation

10

YCB Project

is launched

IROS

Workshop

ICRA

Workshop

2014 2015 2017

Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

Connecting Researchers

• Formed workgroups

─ We got in-touch with people

─ 8 groups are formed in various sub-fields

─ Each group has researchers from different institutions

11 Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

Examples of the Workgroups

12

Within-hand manipulation

Tucker Hermans (U of Texas)

Vikash Kumar (U of Washington)

Alberto Rodriguez (MIT)

Nikhil Narsingh Chavan Dafle (MIT)

Aerial manipulation

Anibal Ollero (University of Seville)

Kostas Alexis (U of Nevada)

Antonio Franchi (CNR)

Force/tactile sensing for manipulation

Nikolaus Correll (U of Colarado)

Rob Howe (Harvard)

Veronica Santos (UCLA)

Matei Ciocarlie (Columbia)

Oliver Brock (TU Berlin)

Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

Connecting Researchers

• Formed workgroups

─ We got in-touch with people

─ 8 groups are formed in various sub-fields

─ Each group has researchers from different institutions

• Designing benchmark with collaboration

─ Reduces the risk of overfitting

─ Improves the dissemination of the benchmark

─ Fosters collaboration

• Organized a workshop to share the results

13 Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

Outline

• Efforts to enable benchmarking in robotic manipulation

14

YCB Project

is launched

IROS

Workshop

ICRA

Workshop

RA-Letters

Special Issue

Announcement

2014 2015 2017 2019 (Feb)

Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

Editing a Special Issue

• A publication/reward mechanism was needed

• A concrete goal was needed

• Editing a special issue in IEEE RA-Letters

─ Asked for a letter of intent

─ Received over 25 letters

─ Connected people working towards the same goal

─ Sent guidelines to both authors and reviewers

─ Sent templates

15 Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

Outline

• Efforts to enable benchmarking in robotic manipulation

16

Benchmarking Tools

• YCB object set

•Associated models

Connecting Researchers

• Forming workgroups

• Organizing workshops

Common Venues

•RA-Letters special issue

•Benchmarking website

YCB Project

is launched

IROS

Workshop

ICRA

Workshop

ICRA

Workshop

RA-Letters

Special Issue

Announcement

2014 2015 2017 2019 (Feb) 2019 (May)

Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

Receiving feedback (ICRA 2019)

• The researchers presented their benchmarks, received feedback 

from other researchers

• Updated the benchmarks before the submission

17 Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

Outline

• Efforts to enable benchmarking in robotic manipulation

18

Benchmarking Tools

• YCB object set

•Associated models

Connecting Researchers

• Forming workgroups

• Organizing workshops

Common Venues

•RA-Letters special issue

•Benchmarking website

YCB Project

is launched

IROS

Workshop

ICRA

Workshop

ICRA

Workshop

RA-Letters

Special Issue

Announcement

2014 2015 2017 2019 (Feb) 2019 (May) 2020 (Jan)

Special Issue 

Publication

Facilitating Benchmarks for Robotic Manipulation – Berk Calli
Worcester Polytechnic Institute

ycbbenchmarks.com

19 Facilitating Benchmarks for Robotic Manipulation
Worcester Polytechnic Institute

Discussion on Future Steps

• Divide into sub-categories

• Identify all the benchmarks out there

─ Have an updated website

─ List of how many groups are using them etc.

─ Leader boards

• Determine the strengths and weaknesses of these existing benchmarks

• We could also do surveys for two separate groups

─ The researchers who have used benchmarks

─ The researchers who have not utilized benchmarks

─ Asking them what can be improved

• We can determine the existing datasets and how they are used in 

literature, as well as some simulation platforms that are utilized

• An RA-Magazine paper introducing the portal?

20 Facilitating Benchmarks for Robotic Manipulation
21 Worcester Polytechnic Institute

Aaron Dollar Berk Calli Sidd Srinivasa Aaron Walsman Pieter Abbeel Adam Norton

Joe Falco Kurt Konolige Jim Bruce Yu Sun Maximo Roa Arjun Singh