# Crowdsourced Evaluation of Robot Programming Environments: Methodology and Application
## 3 Materials and Methods
### Assembly Assembly  
_Assembly_[37] is an open-source, simplified, web-based robotic programming environment developed as part of the CoMeMak project. _Assembly_ UI programming concepts rely on five main building blocks:  
1. Actors (Blocks)  
_Assembly_ implements block-based actors. As visible from Figure 1, most of the basic programming concepts already exist. The first row consists of the actors who are meant for precise movement of the entire robot (move to), adjusting the robot'sgripper position (dp, dr), opening and closing the gripper, and defining a dynamic action. The second row consists of basic conditional statements, such as if, else, elseif, and also loops such as while and repeat. The final row is meant for explicit parameter, variable, and condition setting. _Assembly's_ core idea is to simplify block creation, use, and reuse by standardizing them. Additional help is offered to the end user via the \"help option\" (Figure 2). It can be easily accessed merely by clicking on each actor and appears as follows:  
1 moveTo  
Moves the robot to the current pose, as reflected by the position and rotation parameters  
in the robot control area \(\mathbb{L}\) This actor uses the following parameters:  
* Robot.tep. [x, y, z] \(\rightarrow\) Tool center point (TCP) coordinates in Carthesian space
* Robot.tep. [rx, ry, rz] \(\rightarrow\) TCP rotation around the respective (Euler angles)
* Params.motion. [acceleration, velocity] \(\rightarrow\) Movement acceleration and velocity
* Blackboard  
The blackboard is a behavioral design pattern used in computer programming, which is used to prioritize the actors of different systems that are bound to work either in parallel or sequentially in certain programs [38]. In _Assembly_, this concept is used to ease the creation and management of the actors. As visible from Figure 3, the _Assembly_ blackboard uses the following three objects:  
* \"Robot\" is a read-only object containing the state of the robot, including the joint angles and the coordinates of the end effector. * \"Parameters\" object contains a list of parameters used by the actors from the actor library. Only their values can be modified; the object's name or structure cannot. * \"Variables\" object is editable by end users, and the input is then used for the conditional and loop enabling statements. The \"Parameters\" and \"Variables\" objects can be edited by users using an embedded JSON editor [39]. Figure 1: _Assembly_ actor and task library. Figure 2: _Assembly_ help option. 3. Tasks A workflow can be created by linking a series of actors together. This workflow can then be easily saved via a bookmarking option and subsequently reused as a standalone task or part of another task. Creating a new task is as easy as dragging and dropping necessary actors between the start and stop buttons, automatically triggering the saving of snapshots of the \"Parameters\" and \"Variables\" objects. When loading a task, the content of the global \"Parameters\" and \"Variables\" objects is overwritten by the content of the saved snapshots. When loading a bookmarked task, it will appear as an additional task actor in the actor and task library region. To create compound tasks by reusing, it is essential to load each task individually. The best practice suggests saving an empty task, loading it, and subsequently loading all the other saved tasks within it. This will be the final use case scenario. The intention behind task creation is threefold: 1. [label=()] 2. simplified creation and organization of the robot programs within the browser; 3. reusing tasks via aggregation; 4. sharing with other users and exporting tasks effortlessly as browser bookmarks. 4. Simulator _Assembly_ uses an adapted version of a six degrees of freedom (DOF) robot simulator by [40]. The simulator is embedded in the right-hand side of the _Assembly_ environment. The \"Robot\" object contains the axis angles corresponding to a certain tool center point (TCP) position.