Systems and methods for remote camera control

A camera capable of capturing still images and video and included in a portable media device can be controlled remotely using an accessory. The accessory can register with the PMD to automatically receive notifications whenever there is a change in the camera state. The camera states can include mode, operation status, and configuration settings. The accessory can send instructions to a camera application that interfaces with the camera to control the camera. The accessory can remotely activate the camera, change camera mode, and send instructions to operate the camera. The accessory and the PMD can concurrently control the camera. The PMD can send the captured still images and recorded video to the accessory for preview and receive instructions from the accessory on disposition of the still images and the video.

BACKGROUND

The present disclosure relates generally to media players and in particular to controlling a camera in a portable media device.

A portable media device can store a variety of digital information such as audio tracks, video tracks, photos, and other information. Examples of portable media devices are the iPod® and the iPhone™ portable media devices, which are available from Apple Inc. of Cupertino, Calif.

A portable media device can include one or more connectors or ports that can be used to interface with other devices. For example, the connector or port can enable the portable media device to couple to a host computer, be inserted into a docking system, or receive an accessory device. In the case of the iPod®, e.g., a vast array of accessory devices have been developed that can interconnect to the portable media device.

In some instances, a portable media device can have the capability of taking still photos and recording video. For example, a portable media device may include a camera and an associated camera application to control the camera. The camera application can be used to control various aspects of the camera including taking still photographs and recording video.

In addition to taking pictures and recording video with the camera, an accessory can be used to communicate with the portable media device to control the camera remotely to take pictures and record video.

SUMMARY

Certain embodiments of the present invention relate to remotely controlling a camera included in a portable media device. A portable media device can have ability to take still photos or record video. In some embodiments, an accessory can control the camera by communicating with the portable media device using a plurality of commands. In an embodiment, the portable media device can have a camera application program resident on the portable media device that can communicate with the camera in order to operate the camera.

In some embodiments, the accessory can register with the portable media device to receive notifications regarding camera state changes. The camera state change notifications can be automatically sent to the accessory upon registration. In an embodiment, the camera state change notifications can be sent asynchronously to the accessory whenever there is a change in the camera state. In some embodiments, the accessory can remotely change the camera mode and remotely operate the camera including activating the camera application. In some embodiments, the camera can be concurrently controlled from the accessory and from the portable media device.

In some embodiments, the portable media device can send the captured still image or the recorded video to the accessory for preview. In some embodiments, the accessory can send instructions to the portable media device about the disposition of the still image or the video. In other embodiments, the accessory can access the still images and videos stored in the portable media device remotely and send instructions to the portable media device for playback of the still images, e.g., slide show, and videos.

DETAILED DESCRIPTION

Certain embodiments of the present invention relate to controlling a camera included in a portable media device. A portable media device can have a camera that is capable of capturing still photos and recording video. In an embodiment, the portable media device can have a camera application program resident on the portable media device that can communicate with the camera in order to operate the camera. In some embodiments, an accessory can control the camera using a set of commands exchanged between the portable media device and the accessory. In some embodiments, the accessory can register with the portable media device to receive notifications regarding camera state. Once registered, the accessory can receive notification regarding changes in camera state from the portable media device without having to request an update.

In some embodiments, the accessory can remotely start the camera application if the accessory receives a notification that the camera is inactive. In some embodiments, the accessory can change the camera mode (e.g., between still photo and video modes), instruct the camera to capture a still image, instruct the camera to start video recording, or instruct the camera to stop video recording. An image as described herein can be a still image, e.g., still photo, or a moving image, e.g., a video; hence, “capturing an image” can encompass both still images and video recordings. In some embodiments, the accessory can receive the image captured by the camera for preview and can instruct the portable media device about the disposition of the image, e.g., save or delete. In some embodiments, the accessory can periodically send an image capture command to the portable media device to enable the camera to take a series of still images during a predetermined time. In some embodiments, the accessory can instruct the portable media device to take a series of images over a given time period.

FIG. 1Aillustrates two views of portable media device (PMD)100. PMD100can have the ability to record and store images in addition to the ability to play media tracks and perform other tasks. As illustrated in the front view, PMD100includes display101. As illustrated in the back view, PMD100includes a camera102. Camera102can have the ability to capture still images and record video. In some embodiments, the camera can support video recording in various resolutions, e.g. 640×480, various aspect ratios, e.g., 4:3 or 16:9, and various frame speeds e.g., 25 frames/sec. In some embodiments, the camera can capture still images of various resolutions depending on the image sensor of the camera. Display101can display the images captured by camera102in addition to other information. In some embodiments, PMD100may have a camera application (not shown) installed. The camera application can be implemented as a program that when executed provides a user interface through which a user may control and/or configure the camera. The camera application may accept the user input and control camera102accordingly. For example, the camera application may accept user input regarding changing the mode of camera102between still photo mode and video mode. The camera application may then place camera102in the desired mode and prepare camera102to capture an image. The camera application can also accept user input instructing the camera to capture the image.

In some embodiments, PMD100can communicate with a host computer (not explicitly shown) that executes an image management program. The image management program can allow a user to manage and/or organize the images taken by the camera. The management of the images can include, e.g., storing, deleting, or manipulating an image by subjecting the image to some form of image processing. In some embodiments, PMD100can communicate with accessory150that is capable of controlling camera102.

Accessory150can provide capabilities related to remotely controlling camera102of PMD100, for example, using wireless signals such as infrared, Bluetooth®, or the like. Accessory150can have a display151that is capable of displaying information received from PMD100including a still photo or video recording, in addition to other information. Accessory150can include power button152to turn the accessory on or off, photo button153to instruct camera102of PMD100to capture a still image, record button154to instruct camera102to capture video, mode button155to change the camera mode, and preview button156that can enable the display of an image captured by camera102on accessory display151. In some embodiments, display151can also display various notifications regarding the camera state received from PMD100. In some embodiments, accessory display151can be a touch screen that provides user operable interface elements. In some embodiments, photo button153, record button154, mode button155, and preview button156may be provided as user operable interface elements on a touch screen display. Accessory150can also include a set of playback controls157. Playback controls157can include individual controls for Rewind, Play, Pause, Stop, and Fast-forward. In some embodiments, playback controls157can be provided as user interface elements on a touch screen display. Accessory150may further include an indicator158that can illuminate when the accessory is in operation. For example, when accessory150is in an idle state (not providing instructions to the PMD) indicator158can be in an “off” state. In an embodiment, when accessory150sends an image capture command to PMD100, indicator158can be in an “on” state reverting to the “off” state once the image is captured by PMD100and a notification to that effect is received by accessory150from PMD100. In some embodiments, once accessory150sends a command to PMD100to start recording a video, indicator158can remain illuminated (“on”) for the duration of the recording and revert to the “off” state once the recording is stopped and a notification to that effect is received by accessory150from PMD100. In some embodiments, indicator158can be a light emitting diode (LED).

FIG. 1Billustrates an accessory180according to an embodiment of the present invention. In this embodiment, accessory150can include a power button170, a mode button171, and an action button172. Mode button171can be used to toggle the camera between the still photo mode and video mode. In some embodiments, mode button171is stateless and selecting this button will change the mode of the camera from a current mode to a different mode. For example, if the camera is in a still photo mode and mode button171is selected, the camera can transition to the video mode and vice versa. Action button172can assume various functions depending on the state of the camera. For example, if the camera is in the still photo mode, selecting Action button172can result in the camera taking a still photo, if the camera is recording a video and action button172is selected, the camera can stop recording the video. If the camera is in video mode but not recording video and action button172is selected, the camera can start recording video. Thus, action button172can itself be stateless in that the command communicated by the accessory via action button172can depend on the camera state.

It is to be noted that the PMD and accessory described above are illustrative. PMD100can be any device capable of storing and playing media and capturing still images and/or recording video. In some embodiments, PMD100can record audio along with the video and can have cellular phone capability. In some embodiments, accessory150can have different controls from the ones illustrated inFIG. 1A(e.g., fewer controls or additional controls such as save and delete). In some embodiments, accessory150may not have a display.

FIG. 2is a block diagram of system200according to an embodiment of the present invention. System200can include PMD202(e.g., implementing PMD100ofFIG. 1A) and an accessory220(e.g., implementing accessory150ofFIG. 1A).

Camera207can capture still images and record video. Camera207can be equipped with various capabilities including zoom, focus, aperture control, shutter speed control, and the like. In some embodiments, camera207can be controlled using camera application205, which can be, e.g., program code stored on storage device206and executed by processor204. Camera application205can have a graphical user interface (GUI) associated with it that can provide the capability for a user to control various aspects of the camera. For example, camera application205may provide the capability to adjust the shutter speed for the camera, start and stop video recording, preview, save, or delete stored images, etc. In some embodiments, camera207can be concurrently controlled through user interface208and by accessory220. In some embodiments, camera207may not be operable without first activating camera application205.

Storage device206can be implemented, e.g., using disk, flash memory, or any other non-volatile storage medium. In some embodiments, storage device206can store media assets, such as audio, video, still images, or the like, that can be played by PMD202, including images and videos previously captured by camera207. Storage device206can also store user customized camera settings. Storage device206can store camera application205, which can be used to control or otherwise operate camera207. In some embodiments, storage device206may store a log of all activities related to the camera. For instance, the log can include a list of all instructions received from accessory220along with a time stamp or other identifying information or a list of all notifications sent by the PMD to the accessory.

Storage device206can also store other information such as information about a user's contacts (names, addresses, phone numbers, etc.), scheduled appointments and events, notes, and/or other personal information. In some embodiments, storage device206can store one or more other programs to be executed by processor204(e.g., video game programs, personal information management programs, image processing application, media playback programs, etc.).

User interface208can include input controls such as a touch pad, touch screen, scroll wheel, click wheel, dial, button, keypad, microphone, or the like, as well as output devices such as video screen, indicator lights, speakers, headphone jacks or the like, together with supporting electronics (e.g., digital-to-analog or analog-to-digital converters, signal processors or the like). A user can operate the various input controls of user interface208to invoke the functionality of PMD202and can view and/or hear output from PMD202via user interface208.

Processor204, which can be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), can control the operation of PMD202. For example, in response to user input signals provided by user interface208, processor204can instruct camera207to capture an image. In some embodiments, processor204can also control user interface208to provide interfaces allowing the user to configure various camera settings, e.g., aperture size, shutter speed, using camera application205. In some embodiments, processor204can instruct camera application205to operate camera207to capture an image and send the captured image to the accessory for preview. In some embodiments, processor204can communicate camera state change notifications to the accessory.

Accessory I/O interface214can allow PMD202to communicate with various accessories. For example, accessory I/O interface214might support connections to a remote control unit, an external speaker dock, an external video device, or the like. In some embodiments, accessory I/O interface214includes a 30-pin connector corresponding to the connector used on iPod® products manufactured and sold by Apple Inc. Alternatively or additionally, accessory I/O interface214can include a different connector and/or wireless interface (e.g., Bluetooth or the like).

Accessory220can include controller224, user interface222, PMD I/O interface226, cache228, and audio output device230. Controller224can include, e.g., a microprocessor or microcontroller executing program code to perform various functions such as controlling camera207, displaying images captured by camera207, receiving notifications about camera state change from PMD202, and the like. User interface222can include input controls such as a touch pad, touch screen, scroll wheel, click wheel, dial, button, keypad, microphone, or the like, as well as output devices such as video screen, indicator lights, or the like, together with supporting electronics (e.g., digital-to-analog or analog-to-digital converters, signal processors or the like). A user can operate the various input controls of user interface222to invoke the functionality of accessory220and can view output from accessory220via user interface222and hear output via audio output device230. In addition, in some embodiments, a user can operate PMD202via user interface222. PMD I/O interface226can allow accessory220to communicate with PMD202(or another PMD). Examples are described below.

Cache228, which can be implemented using volatile and/or nonvolatile memory, provides storage for various information including information obtained from PMD202. For example, in some embodiments, accessory220can obtain notifications related to camera207and images captured by camera207from PMD202. Any or all of this information can be stored in cache228. Accessory220can be a simple remote control without video display, audio output, or cache.

Audio output device230can include one or more speakers or driver circuits and connectors for external speakers, thereby enabling audio to be presented to a user. In some embodiments, controller224can receive an audiovisual recording from PMD202via PMD I/O interface226and can provide the audio signals associated with the video with or without further processing to audio output device230. Audio output device230can transform the signals as appropriate for presentation to the user.

It will be appreciated that the system configurations and components described herein are illustrative and that variations and modifications are possible. The PMD and/or accessory can have other capabilities not specifically described herein. For example, PMD202can play stored media such as songs, audio books, videos, including media obtained from sources other than camera207. In some embodiments, the accessory can be used to control a camera in the PMD remotely and operate the camera to capture still photos or videos. For example, the accessory can send various commands to the PMD in order to operate the camera and can receive camera related notifications from the PMD. In some embodiments, the camera can be concurrently operated using the accessory and the PMD.

Accessory I/O interface214of PMD202and PMD I/O interface226of accessory220allow PMD202to be connected to accessory220and subsequently disconnected from accessory220. As used herein, PMD202and accessory220are “connected” whenever a communication channel between accessory I/O interface214and PMD I/O interface226is open and are “disconnected” whenever the communication channel is closed. Connection can be achieved by physical attachment (e.g., between respective mating connectors of PMD202and accessory220), by an indirect connection such as a cable, or by establishing a wireless communication channel. Similarly, disconnection can be achieved by physical detachment, disconnecting a cable, powering down accessory220or PMD202, or closing the wireless communication channel. Thus, a variety of communication channels can be used, including wired channels such as USB, FireWire, or universal asynchronous receiver/transmitter (“UART”), or wireless channels such as Bluetooth and Wi-Fi. In some embodiments, multiple communication channels between a PMD and an accessory can be open concurrently, or a PMD can be connected to multiple accessories, with each accessory using a different communication channel.

Regardless of the particular communication channel, as long as PMD202and accessory220are connected to each other, the devices can communicate by exchanging commands and data according to a PMD-specific protocol. The PMD specific protocol can define a format for sending messages between PMD202and accessory220. For instance, the specific protocol can specify that each message be sent in a packet with a header and an optional payload. The header provides basic information (e.g., a start indicator, length of the packet, and a command to be processed by the recipient), while the payload provides any data associated with the command; the amount of associated data can be different for different commands, and some commands can provide for variable-length payloads. In various embodiments, the PMD-specific protocol can define specific commands to indicate an action to be taken by the recipient; to signal completion of a task, change of state, or occurrence of an error; and/or to identify the nature of the associated data. In some embodiments, the commands can be defined such that a particular command is valid in only one direction. The packet can also include error-detection or error-correction codes as known in the art.

The PMD-specific protocol can define a number of “lingoes,” where a “lingo” is a group of related commands that can be supported (or unsupported) by various classes of accessories. In some embodiments, a command can be uniquely identified by a first byte identifying the lingo to which the command belongs and a second byte identifying the particular command within the lingo. Other command structures can also be used. It is not required that all accessories, or all PMDs to which an accessory can be connected, support every lingo defined within the PMD-specific protocol.

In some embodiments, every accessory220and every PMD202that are designed to be interoperable with each other support at least a “general” lingo that includes commands common to all such devices. The general lingo can include commands enabling the PMD and the accessory to identify themselves to each other and to provide general information about their respective capabilities, including which (if any) other lingoes each supports. The general lingo can also include authentication commands that the PMD can use to verify the purported identity and capabilities of the accessory (or vice versa), and the accessory (or PMD) can be blocked from invoking certain commands or lingoes if the authentication is unsuccessful.

In some embodiments the general lingo can also provide a notification capability. For example, PMD202can generate notifications in response to various events that change the status of PMD202, such as launching or exiting various applications (e.g., camera application205), changing state within an application (e.g., changing between still-photo and video modes), and so on. Accessory220, when connected to PMD202, can “register” to receive all notifications or selected classes of notifications by sending a registration command to PMD202; an example is described below. Once accessory220has registered for a particular class (or classes) of notifications, PMD202automatically begins to send notifications to accessory220whenever any event within the registered class(es) occurs. Notification conveniently allows accessory220to maintain current information about the status of PMD202without having to send requests for status information. In some embodiments, the notifications about the changes in camera state can be communicated to the accessory asynchronously. Camera207can change states independent of accessory220and accessory can get notified of the change in state asynchronously.

A PMD-specific protocol can also include various other lingoes, such as a simple remote lingo that allows accessory220to send a command indicating a function of PMD202to be invoked, a remote user interface lingo that can be used to communicate commands and data related to replicating all or part of a user interface of PMD202on accessory220(thereby supporting a more advanced remote control), a camera lingo that allows accessory220to control various settings of camera207, a storage lingo that allows accessory220to store data on PMD202, and so on. Any lingo or combination of lingoes or other commands or groups of commands can be used in connection with a PMD-specific protocol.

PMD State Diagram

FIG. 3is a simplified state diagram300for PMD202illustrating aspects of operation of camera207according to an embodiment of the present invention. When PMD202is turned on or when the camera is inactive, PMD202can be in a Camera not Ready state301. When a “CameraON” event occurs, PMD202transitions to either Photo Preview state302or video preview state304. In some embodiments, PMD202may remember the last camera state prior to exiting camera operation and transition to that camera state, e.g., video preview, on occurrence of the CameraON event. The CameraON event can occur in response to any user input indicating that the user desires to start the camera. For example, the user may activate a camera icon on a graphical user interface of PMD202or operate a control of accessory220that causes accessory220to send a command to PMD202to launch camera application207.

In Photo Preview state302, PMD202can receive input for operating the camera to take a still image, change the camera settings, e.g., focus, zoom, etc., review previously captured images, change camera mode, and the like. In some embodiments, PMD202can remain in the Photo Preview state302until the camera application quits, e.g., “CameraOFF” event, or PMD202receives a command to change camera mode. In the event that PMD202receives a command to capture a still image, e.g., “CameraAction” event, PMD202transitions to Photo Taken state303to capture the current image from the camera sensor into memory or other storage medium. In some embodiments, upon capturing the still image, PMD202returns to Photo Preview state302.

In some embodiments, when PMD202is in Photo Preview state304, it can receive a command to change the camera mode, resulting in a “ToggleMode” event. The ToggleMode event can occur in response to a command received from accessory220or via the user interface of PMD202to change the camera mode. Upon occurrence of the ToggleMode event, PMD202transitions to Video Preview state304. Similarly, if PMD202is in Video Preview state304and the ToggleMode event occurs, PMD202transitions to Photo Preview state302.

In Video Preview state304, PMD202can receive input for operating the camera to capture video, changing camera settings, changing camera mode, or turning the camera off. If PMD202receives a command to capture video, e.g., CameraAction event, PMD202transitions to Video Recording state305. PMD202can stay in Video Recording state305until it receives a command to stop recording the video, e.g., CameraAction event, or camera application205is turned off, e.g., CameraOFF event.

It will be appreciated that the various states described in reference toFIG. 3are for illustrative purposes only. Once skilled in the art will realize that various other states are possible depending on the PMD capabilities.

Accessory State Diagram

FIG. 4is a simplified state diagram400for accessory220illustrating aspects of operation of camera207and its relation to the accessory states, according to an embodiment of the present invention. When accessory220is powered on, reset, or initially connected to PMD202, it can be in Photo with Indicator OFF state401. In some embodiments, accessory220can remember the state it was in just prior to last power down, reset, etc. and transition to that state upon power on or reset. In the embodiment illustrated inFIG. 4, Photo with Indicator OFF state401is designated as the “default” state for accessory220for ease of explanation. Accessory220can transition to any of the other states from this default state upon receiving notification of occurrence of the appropriate event from PMD202. It should be understood that accessory220can be programmed to be in a different state upon power up and/or reset.

Accessory220can transition to different states in response to notifications of events from PMD202. For example, in Photo with Indicator OFF state401, if accessory220receives a notification from PMD202that camera207has transitioned to the Photo Taken state, accessory220can transition to Photo with Indicator ON state404. In some embodiments, the indicator on accessory220may flash briefly or turn on as the accessory transitions to Photo with Indicator ON state404. Once camera207captures the still image and notifies the accessory that it has transitioned back to photo preview state, accessory220can transition back to Photo with Indicator OFF state401; this transition can include turning off the indicator. In some embodiments, the transitions from Photo with Indicator OFF state401to Photo with Indicator ON state404and back again may occur quickly so that the indicator flashes briefly.

If, while accessory220is in Photo with Indicator OFF state401, it receives notification that camera207has changed its state to Video Preview, accessory220can transition to Video with Indicator OFF state402. As described above, camera207may change its state based on instructions received from accessory220or PMD202. In the Video with Indicator OFF state402, accessory220can receive a notification from PMD202that may cause accessory220to change to Photo with Indicator OFF state401, e.g., camera mode change or camera off notification. In addition, while in Video with Indicator OFF state402, if accessory220receives notification that camera207has begun to capture a video, e.g., video recording notification, accessory220can transition to Video with Indicator ON state405. In some embodiments, accessory220can transition directly to Video with Indicator ON state405from Photo with Indicator OFF state401upon receiving an appropriate notification from PMD202.

It will be appreciated that the various states described in reference toFIGS. 3 and 4are for illustrative purposes only. One skilled in the art will realize that various other states are possible depending on the accessory capabilities.

In some embodiments, accessory220can be stateless in that the accessory may not have states that correspond to the states of the camera. For example, the camera may change states as illustrated inFIG. 3, but there may not be any corresponding state changes for the accessory. The commands sent by the accessory will operate the camera based on the state of the camera without the accessory providing specific commands. For example, if the camera is in a photo preview state, pressing the action button on accessory150ofFIG. 1Bcan result in the camera capturing an image without the accessory specifically instructing the camera to capture an image. The accessory can operate independent of the camera state. If the camera is in photo preview state and the mode button on the accessory is selected, the accessory can send a command to the camera to change modes without knowing the current camera mode or specifying a particular new mode for the camera. In some embodiments, the camera can change states without any instructions from the accessory, e.g., by receiving instructions from the PMD user interface.

FIG. 5shows table500listing commands that can be used to control a camera included in a PMD according to an embodiment of the present invention. These commands can be incorporated into a camera lingo (or other group of commands) within a communication protocol supported by the PMD and the accessory, thereby allowing the PMD and the accessory to exchange information related to the camera.

The SetEventNotification command can be sent by accessory220to PMD202to register for notifications of various changes to the PMD's state. In one embodiment, the payload of the SetEventNotification command includes a bitmask in which each bit is associated with a different class of notifications. For example, if PMD202has multiple functions, the notifications can be grouped into classes according to function (e.g., camera, stored media playback, telephony, data network access, navigation, etc.), and accessory220can register for classes that will affect its operation. Thus, for example, an accessory that can control camera functionality can register for camera notifications without receiving, e.g., telephony or navigation notifications. In some embodiments, PMD202can return an acknowledgement (not explicitly shown inFIG. 3) to confirm receipt of the SetEventNotification command.

The PMDNotification command can be sent by PMD202to accessory220to notify accessory220of a state transition or other event that may affect operation of the accessory. The payload can include a notification message indicating the type of event (e.g., using a byte code) and optionally other information that depends on the type of event. In some embodiments, PMD202sends notifications only for events in a class for which the accessory has registered using the SetEventNotification command. For example, in one embodiment, there can be defined a “camera” class of notifications. The camera class can include notifications corresponding to, e.g., the CameraON and CameraOFF events ofFIG. 3, as well as notifications related to operation of camera such as capturing a still image or starting and stopping a video recording. In particular, in some embodiments, notifications can be sent for each CameraAction event shown inFIG. 3.

The CameraButtonStatus command can be sent from the accessory to the PMD to provide an instruction for operating the camera. The payload of this command can include a bit mask in which different bits map to different camera operations. For example, one bit may map to a camera mode toggle between video and still photo modes and another bit may map to a Camera Action operation. The command can be used to send the desired instruction by setting the corresponding bit(s) in the bitmask. For instance, the CameraButtonStatus command can be used to toggle the camera mode between Photo Preview and Video Preview by setting the appropriate bit in the bit mask. In some embodiments, this command can be used to instruct the camera to capture a still photo or record a video depending on the mode of the camera. For instance, if a user of the accessory desires to capture a still photo but the PMDNotification command indicates that the camera is in a Video Preview mode, the user can instruct the accessory to send the CameraButtonStatus command with the appropriate bit set to toggle the camera mode to Photo Preview mode. Once the PMD confirms the change in mode via the PMDNotification command, the accessory can send the CameraButtonStatus command with the bit corresponding to Camera Action being set. The camera can then capture a still image. A single Camera Action bit of the CameraButtonStatus command can be associated with multiple camera operations such as taking still photo if the camera is in Photo Preview mode, starting video recording if the camera is in Video Preview mode, and stopping video recording if the camera is in Video Recording mode.

It will be appreciated that the commands described herein are illustrative and that variations and modifications are possible. For instance, additional commands can be used to facilitate communication of camera state information and operation instructions between the PMD and the accessory. For example, the accessory can send a GetEventNotification command to the PMD to confirm the event classes for which the accessory is currently registered to receive notifications. The PMD can respond with a RetEventNotification command informing the accessory about the event classes that the accessory is currently registered for and if the currently registered event classes are not the ones the accessory wishes to remain registered for, the subscription of the event classes can be changed, e.g., using the SetEventNotification command. In some embodiments, the accessory may query the PMD about all the event classes that the PMD supports for providing notifications, e.g., using a GetSupportedEventNotification command. The PMD may respond with a RetSupportedEventNotification command. The payload of the RetSupportedEventNotification command can include a bitmask indicating all possible event classes for which the PMD can generate notifications. In some embodiments, each bit of the bitmask can correspond to an event or an event class. Further, not all of the commands described herein need be supported, and other protocols can be used.

In some embodiments, PMD202can generate camera state notifications and communicate the notifications to the accessory. In some embodiments, these notifications are provided to the accessory asynchronously, i.e., without the accessory requesting the notification. In some embodiments, before PMD202can send the camera state notifications, the accessory may have to register with PMD202, e.g., using the SetEventNotification command, to receive the notifications. Once the accessory registers with PMD202, it can automatically receive notifications about changes in the state of the camera. Camera state notifications can include information about, camera mode (photo or video), camera operation (video recording in progress, photo taken, etc.), camera status (active (on) or inactive (off)), whether the camera application is currently running or not, etc. Every time there is change in the state of the camera, PMD202can send a notification to the accessory. In some embodiments, a notification regarding camera state may be sent to the accessory even if a command to change the camera state does not result in an actual change of state of the camera. In addition, in some embodiments additional commands can be provided to allow accessory220to direct camera application205to control various settings of the camera including, zoom, shutter speed, aperture size, flash, focus, and the like.

FIG. 6is a flow diagram of a process600that can be used by an accessory to control a camera included in a portable media device according to an embodiment of the present invention. Process600can be performed, e.g., by accessory220ofFIG. 2. At block601, the PMD can authenticate and identify the accessory, e.g., using one or more commands from the general lingo described above. Once the accessory is authenticated, it can exchange information with the PMD. In some embodiments, during identification at block601, accessory220can specify a preferred initial state for PMD202(e.g., whether PMD202should enter camera mode), and block601can also include receiving confirmation that the preferred state has been entered.

At block602, the accessory can register with the PMD for receiving camera state notifications e.g., by sending the SetEventNotification command with appropriate parameters. At block603, the accessory can receive a camera state change notification from the PMD indicating the current state of the camera e.g., via the PMDNotification command. At block604, the accessory can analyze the received notification to check whether the camera application is currently active. If it is determined that the camera application is not currently active, the accessory can wait at block611until another notification is received and process600returns to block603. In an alternative embodiment, if at block604, it is determined that the camera application is not active, the accessory can send a command to the PMD to activate the camera application, e.g., by using the CameraButtonStatus command. In some embodiments, this command can be sent automatically by the accessory or in response to user input requiring camera activation. Once it is determined, at block604, that the camera application is active, the accessory can receive current camera mode information at block605e.g., via the PMDNotification command.

Once the current camera mode is received by the accessory, the accessory can present the current mode information at block606, e.g., on its display, and wait for further input. At block607, the accessory can receive input, e.g., from a user operating the accessory. Once the accessory receives an input, the accessory can check, at block608, to determine if the input corresponds to a request to change the camera mode. If it is determined that the input is not to change the camera mode, processing can proceed to block609. If it is determined that the input is a request to change camera mode, the accessory can send instructions to the PMD to change the camera mode at block612, e.g., using the CameraButtonStatus command. In some embodiments, block612can include receiving notification of the new camera mode. Once the accessory sends the instruction to change camera mode at block612, process600can return to block605where the PMD can receive notification of the camera mode change and then await further inputs at block607.

At block609, the accessory can check whether the received input corresponds to an instruction for operating the camera. If it is determined that the received input corresponds to an instruction for operating the camera, the accessory can send instructions to the PMD to operate the camera at block610, e.g., using the CameraButtonStatus command. For example, the instructions to operate the camera can include instructions to change camera settings or instructions to capture an image. After sending the instructions to the PMD and receiving response from the PMD, process600can return to block607to await further input. If at block609, it is determined that the received input is not to operate the camera, process600can return to block607and the accessory can check whether it has received any new input. Process600can continue until the accessory is disconnected from the PMD. In some embodiments, at block607, instead of a user input, the input received can be a notification regarding change in the camera state. For example, if a user instructs the camera to capture an image by operating the user interface of the PMD after the accessory has received the camera mode information, e.g., Photo Preview, at block605, the input received at block607can be the new camera state e.g., Photo Taken.

It will be appreciated that process600described herein is illustrative and that variations and modifications are possible. Acts described as sequential can be executed in parallel, order of acts can be varied, and acts can be modified or combined. For instance, the accessory may send the command to change the camera mode to a desired mode before the PMD sends a notification about the current camera mode. In some embodiments, the accessory may not wait for the PMD to confirm a mode change before sending the instructions to operate the camera. In some embodiments, after the camera captures an image, the PMD may send the picture to the accessory for display on the accessory either automatically or in response to an image request command sent by the accessory. In some embodiments, the accessory may provide disposition instructions for the image to the PMD once it is previewed, e.g., store the image, delete the image, etc.

FIG. 7is a flow diagram of a process700that can be used by a portable media device to communicate with an accessory to control a camera included in the portable media device according to an embodiment of the present invention. Process700can be performed, e.g., by PMD202ofFIG. 2. At block701, the PMD can identify and authenticate the accessory, e.g., by using the commands in the general lingo. Once authenticated, the accessory can exchange information with the PMD. In some embodiments, accessory220can indicate that PMD202should initialize in the camera ON mode, e.g., during identification at block701.

At block702, the PMD can receive a request from the accessory to be registered to receive camera notifications, e.g., by using the SetEventNotification command. In some embodiments, the PMD may send an acknowledgement command to indicate that the request has been successfully received. In some embodiments, the PMD can also send current camera state information to the accessory, e.g., via the PMDNotification command, as a reply to the SetEventNotification command. At block703, the PMD can verify whether the camera application is currently active. If it is determined that the camera application is not currently active, process700can wait for the camera application to be started at block710. In some embodiments, the PMD can generate a notification indicating that the camera application is currently off. In some embodiments, a user operating the PMD through its user interface can start the camera application. In an alternative embodiment, the accessory can remotely start the camera application.

If it is determined at block703that the camera application is currently active, or upon activation of the camera application, the PMD can send a camera state notification to the accessory, at block704, indicating the change in status of the camera application. At block705, the PMD may send a notification to the accessory indicating the current camera mode, e.g., still photo or video. In some embodiments, the camera application controls the mode selection and reports the mode to the PMD's operating system to facilitate notification to the accessory. At block706, the PMD can check whether it has received any instructions from the accessory to operate the camera. As described above, in some embodiments, the camera can be concurrently controlled by the accessory and the PMD. If it is determined at block706that PMD has received instructions from the accessory, the PMD can check, at block711, whether the instructions are to exit the camera application. If the instructions are to exit the camera application, process700ends. If the instructions do not correspond to exiting the camera application, the PMD can execute the instructions and operate the camera accordingly at block708. In some embodiments, the instructions can be to change the mode of the camera by toggling between the still photo and video modes. In some embodiments, the instructions from the accessory can be to capture an image using the current mode of the camera. In yet other embodiments, the instructions can include adjustments to camera settings such as zoom, focus, aperture size, or shutter speed.

If it is determined at block706that no instructions have been received from the accessory, the PMD can check whether it has received instructions for operating the camera through the PMD user interface at block707. If it is determined that instructions have been received through the PMD user interface, the PMD can operate the camera according to the instructions, at block708. As described above, the instructions can include instructions for capturing an image, instructions for adjusting camera settings, or instructions for disposition of the images previously captured. In some embodiments, process700can check, at block711, whether the instructions received from the PMD user interface are to exit the camera application and proceed accordingly if the instructions are to exit the camera application. If the instructions are not to exit the camera application, PMD can operate the camera based on the instructions at block708. If it is determined, that no instructions are received at block707, process700can wait to receive instructions from the accessory or through the PMD user interface and the process can return to block706. Upon executing the instructions received from either the accessory or the PMD user interface, the PMD can communicate the results of executing the instructions to the accessory and/or the PMD user interface at block709. In some embodiments, the PMD can send any images captured by the camera to the accessory for preview by the accessory, either automatically or in response to requests from the accessory. The accessory may then send instructions regarding disposition of the image to the PMD.

It will be appreciated that process700described herein is illustrative and that variations and modifications are possible. Acts described as sequential can be executed in parallel, order of acts can be varied, and acts can be modified or combined. For instance, the PMD may send a camera notification along with the acknowledgement message for the request received at block702instead of sending the camera notification as a separate message.

It is to be noted thatFIGS. 6 and 7illustrate one possible sequence of events for controlling the camera by the accessory and the PMD, respectively and include sample camera state transitions that can occur during the respective processes. One skilled in the art will realize that other methods for camera control are possible that may include different camera state transitions and/or sequences of transitions than the ones illustrated.

Embodiments described herein provide for control of a camera included in a PMD. In some embodiments, the accessory may register with the PMD to receive camera notifications. Once registered, the accessory can automatically receive camera notifications every time there is a change in the camera state, without having to query the PMD for the current state. In some embodiments, the notifications can include camera status change (e.g., active or inactive), camera mode change (e.g., still photo or video), or camera control information (focus, zoom, aperture size, etc.). In some embodiments, the accessory can remotely activate the camera by starting a camera application on the PMD that controls the camera, remotely change the camera mode, and remotely provide instructions to operate the camera. In some embodiments, the accessory and the PMD can concurrently control the camera, i.e., both the PMD and the accessory can send instructions to operate the camera concurrently. For example, a user can start video recording by operating the PMD user interface and stop the video recording using the accessory, a user can set the camera mode from the accessory and take a photo/start recording from the PMD user interface. In some embodiments, the PMD may not allow conflicting commands to be sent to the camera. For example, if the accessory attempts to change the camera mode to still mode while the camera is recording a video, the mode change command can be blocked by the PMD and the accessory notified accordingly. In some embodiments, the PMD may send the image captured by the camera to the accessory for viewing purposes. In some embodiments, the accessory may instruct the PMD about disposition of images captured and/or stored on the PMD.

In some embodiments, the camera can be external to the PMD while the camera application can be resident on the PMD. In this instance, the camera can still be controlled using the accessory via the camera application using the above-mentioned methods. In some embodiments, the external camera can be connected to the accessory I/O interface or a different interface. A camera application resident on the PMD can be operated to send control signals to the external camera in order to operate the camera. In some embodiments, the external camera can be concurrently controlled using the accessory and the PMD.

While the invention has been described with respect to specific embodiments, those skilled in the art will recognize that numerous modifications are possible. For instance, the accessory can send instructions to the PMD to instruct the PMD to present a slide-show of the images stored on the PMD. In some embodiments, the accessory can store images captured by the camera in its cache and these images can be shared with other accessories and/or PMDs. In some embodiments, the accessory can be programmed to send the CameraButtonStatus command to the PMD periodically, e.g., every 1 sec, for capturing a series of still images over a predetermined period. In some embodiments, the accessory can send a single instruction to the PMD to capture a series of images at a certain frequency over a given period, e.g., capture an image every 200 seconds over a period of 1 hour. In some embodiments, the PMD may have an illuminating device, e.g., a light emitting diode (LED) that can be used to provide additional lighting in low light environments to facilitate image capture. The illuminating device may work in conjunction with or in lieu of a flash that may be included in the PMD. In some embodiments, the illuminating device can be controlled by the accessory, e.g., by sending a CameraButtonStatus command with a bitmask bit set to toggle the illuminating device on or off.

In addition, embodiments of the present invention can be realized using any combination of dedicated components and/or programmable processors and/or other programmable devices. While the embodiments described above can make reference to specific hardware and software components, those skilled in the art will appreciate that different combinations of hardware and/or software components can also be used and that particular operations described as being implemented in hardware might also be implemented in software or vice versa.

Computer programs incorporating various features of the present invention can be encoded on various computer readable media for storage and/or transmission; suitable media include magnetic disk or tape, optical storage media such as compact disk (CD) or DVD (digital versatile disk), flash memory, and the like. Computer readable media encoded with the program code can be packaged with a compatible device or provided separately from other devices (e.g., via Internet download).