A computer program comprising computer readable code which, when executed by a computer of a teleoperated surgical system having an imaging device, causes the computer to carry out a method controlling the teleoperated surgical system, the method comprising: for each of a multiplicity of occurrences of a surgical procedure within one or more instances of the surgical system, recording surgical instrument kinematic information, including respective multi-dimensional vectors representing kinematic features of one or more of instantaneous velocity, instantaneous acceleration and instantaneous three-dimensional position of the surgical instrument, indicative of surgical instrument motion produced within the surgical system during the occurrence of the surgical procedure and recording images from the imaging device of a surgical scene within the surgical system during the occurrence of the surgical procedure; determining respective kinematic signatures (SigKi) associated with respective surgical instrument motions; determining respective surgical image signatures, including respective multi-dimensional vectors representing image features of one or more of color, texture, and geometry, associated with respective images of surgical scenes; producing an information structure in a computer readable storage device that associates respective surgical image signatures (SigIi) and kinematic signatures (SigKi) with respective control signals; receiving system actuation state information generated during a performance of the surgical procedure, wherein the system actuation state information comprises docked, following and head-in; identifying a control signal rules information structure which includes one or more rules to associate surgical instrument kinematic signatures, surgical image signatures and system actuation state information with control signals; comparing (1504, 1514), during the performance of the surgical procedure within an instance of the surgical system, surgical instrument kinematic information generated based upon the actuation commands during the performance with at least one respective kinematic signature (SigKi) and surgical images within a surgical scene within the instance of the surgical system with at least one respective surgical image signature (SigIi); determining that the surgical instrument kinematic signature, surgical image signatures and the system actuation state information matches the one or more rules; and launching, during the performance of the surgical procedure within the instance of the surgical system, an associated respective control signal of the one or more rules within the surgical system in response to the match. for each of a multiplicity of occurrences of a surgical procedure within one or more instances of the surgical system, recording surgical instrument kinematic information, including respective multi-dimensional vectors representing kinematic features of one or more of instantaneous velocity, instantaneous acceleration and instantaneous three-dimensional position of the surgical instrument, indicative of surgical instrument motion produced within the surgical system during the occurrence of the surgical procedure and recording images from the imaging device of a surgical scene within the surgical system during the occurrence of the surgical procedure; determining respective kinematic signatures (SigKi) associated with respective surgical instrument motions; determining respective surgical image signatures, including respective multi-dimensional vectors representing image features of one or more of color, texture, and geometry, associated with respective images of surgical scenes; producing an information structure in a computer readable storage device that associates respective surgical image signatures (SigIi) and kinematic signatures (SigKi) with respective control signals; receiving system actuation state information generated during a performance of the surgical procedure, wherein the system actuation state information comprises docked, following and head-in; identifying a control signal rules information structure which includes one or more rules to associate surgical instrument kinematic signatures, surgical image signatures and system actuation state information with control signals; comparing (1504, 1514), during the performance of the surgical procedure within an instance of the surgical system, surgical instrument kinematic information generated based upon the actuation commands during the performance with at least one respective kinematic signature (SigKi) and surgical images within a surgical scene within the instance of the surgical system with at least one respective surgical image signature (SigIi); determining that the surgical instrument kinematic signature, surgical image signatures and the system actuation state information matches the one or more rules; and launching, during the performance of the surgical procedure within the instance of the surgical system, an associated respective control signal of the one or more rules within the surgical system in response to the match.