A control device that generates a control command for controlling an operation of a robot device that operates in an environment where a plurality of objects (37, W1, W2) is present, the objects including end effector of the robotic device, a workpiece (W1, W2) and an obstacle, the control device comprising: a target setting unit adapted to set a relative relationship amount (rn) between the plurality of objects serving as a final target (gn), the relative relationship amount (r0, r1, r2, rn) indicating an attribute related to a relative and physical relationship between the plurality of objects (37, W1, W2) in the environment, the relative relationship amount includes an amount that respectively represents relative positions and relative postures between each of the objects in the environment; a data acquisition unit adapted to repeatedly acquire observation data from a sensor that observes the plurality of objects (37, W1, W2) present in the environment; a relationship identifying unit adapted to repeatedly calculate a relative relationship amount (p1 to pn) between the plurality of objects (37, W1, W2) from the observation data being acquired; an action determination unit adapted to determine a series of relative relationship amounts (r1, r2, rn) in a target state (g1, g2, gn) of the plurality of objects (37, W1, W2) until the relative relationship amount (rn) of the final target (gn) being set is realized from the relative relationship amount between the plurality of objects at a starting time point (st) of controlling the operation; a command determination unit adapted to repeatedly determine a control command (411, 412, 41n) to be provided to the robot device so that the relative relationship amount (p1, p2, pn) in a current state (n1, n2, nn) calculated from latest observation data acquired from the sensor is changed to the relative relationship amount (r1, r2, rn) in a state of a next transition target (g1, g2, gn) of the current state (n1, n2, nn), included in the series of relative relationship amounts (rn) until the relative relationship amount (rn) of the final target is realized; and an output unit (106, 106A) that outputs the control command (411, 412, 41n) being determined to the robot device. a target setting unit adapted to set a relative relationship amount (rn) between the plurality of objects serving as a final target (gn), the relative relationship amount (r0, r1, r2, rn) indicating an attribute related to a relative and physical relationship between the plurality of objects (37, W1, W2) in the environment, the relative relationship amount includes an amount that respectively represents relative positions and relative postures between each of the objects in the environment; a data acquisition unit adapted to repeatedly acquire observation data from a sensor that observes the plurality of objects (37, W1, W2) present in the environment; a relationship identifying unit adapted to repeatedly calculate a relative relationship amount (p1 to pn) between the plurality of objects (37, W1, W2) from the observation data being acquired; an action determination unit adapted to determine a series of relative relationship amounts (r1, r2, rn) in a target state (g1, g2, gn) of the plurality of objects (37, W1, W2) until the relative relationship amount (rn) of the final target (gn) being set is realized from the relative relationship amount between the plurality of objects at a starting time point (st) of controlling the operation; a command determination unit adapted to repeatedly determine a control command (411, 412, 41n) to be provided to the robot device so that the relative relationship amount (p1, p2, pn) in a current state (n1, n2, nn) calculated from latest observation data acquired from the sensor is changed to the relative relationship amount (r1, r2, rn) in a state of a next transition target (g1, g2, gn) of the current state (n1, n2, nn), included in the series of relative relationship amounts (rn) until the relative relationship amount (rn) of the final target is realized; and an output unit (106, 106A) that outputs the control command (411, 412, 41n) being determined to the robot device.