Data transmission

Audio-visual data is provided to a network-connected terminal. First audio-visual data containing first information encoded at a first data rate and second audio-visual data containing first information encoded at a second data rate that is more than the first data rate are received. The first audio-visual data is provided to the network-connected terminal. A condition is identified to the effect that the amount of data being sent to the terminal is less than the available bandwidth. The difference between the data rates of the first and second audio-visual data is calculated, additional data packets are sent to the terminal to determine whether the available bandwidth is large enough to provide the first audio-visual data to the terminal; and if it is determined that the available bandwidth is large enough, provision of the first audio-visual data to the terminal is stopped and the second audio-visual data is provided instead.

TECHNICAL FIELD

The present invention relates to a method of providing data over a network.

BACKGROUND OF THE INVENTION

It is possible to wirelessly provide audio-visual data such as television channels, often via the Internet, to mobile devices such as mobile telephones. However, because available bandwidth frequently fluctuates it is generally necessary to send data representing a low quality of picture and sound in order to avoid high packet loss and a degraded user experience.

BRIEF SUMMARY OF THE INVENTION

According to an aspect of the present invention, there is provided a method of providing audio-visual data over a network, comprising the steps of receiving first audio-visual data containing first information encoded at a first data rate, and second audio-visual data containing first information encoded at a second data rate that is higher than the first data rate; providing the first audio-visual data to a network-connected terminal; identifying a condition to the effect that the amount of data being sent to the terminal is less than the available bandwidth; calculating the difference between the data rates of the first and second audio-visual data; sending additional data packets to the terminal to determine whether the available bandwidth is large enough to provide the first audio-visual data to the terminal; and if it is determined that the available bandwidth is large enough, stopping provision of the first audio-visual data to the terminal and providing the second audio-visual data to the terminal.

DESCRIPTION OF THE BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 1illustrates a networked environment in which the invention may be used. Terminals101,102,103,104,105,106and107receive data via the Internet108. The terminals are typically mobile telephones or Personal Digital Assistants (PDAs) but could be any computing device capable of connecting to the Internet, including a home or office computer.

The data is provided over a variety of networks, including in this example radio networks such as mobile telephony networks or wireless networks. A Third Generation (3G) mobile telephony network, connected to the Internet108, includes a gateway109which provides connectivity to a network of base stations. Mobile telephones102and103are each connected to one of these base stations. A General Packet Radio Service (GPRS) gateway110is connected to the Internet108and provides connection to a network of GPRS base stations. PDAs104and105are each connected to one of these stations. A GSM gateway111is connected to the Internet108, providing connectivity for mobile telephone101. Internet Service Provider (ISP)112is connected to the Internet108and provides internet access for PC106and a Wireless Network or Wireless Fidelity (WiFi) gateway113. PDA107has a link to gateway113. Thus there is a number of ways in which a terminal may link to the Internet108in order to receive data.

The data received by terminals101to107is provided by content server114and media server115. Content server114provides many kinds of data required by users of terminals, for example news and sports, financial data, maps and telephone directories, television and cinema listings, and so on. Some of this data may be available to subscribers and some may be free. Users may make transactions with content server114such as purchasing stocks or placing bets, changing their subscription levels or personal details, and so on.

Media server115provides streamed audio-visual data such as television channels, media-on-demand or downloadable music videos. Streamed television channels are provided to media server115by Real Time Streaming Protocol (RTSP) servers116,117and118.

FIG. 2details PDA104. As described above, this is an example of a terminal that could be used in a system embodying the invention. It includes a CPU201with a clock speed of 400 megahertz (MHz) with memory202being provided by 64 megabytes (MB) of RAM. 256 MB of non-volatile FLASH memory203is provided for program and data storage. Liquid crystal display204is used to display information to the user. Input/output205processes the input of the keys and buttons513while audio input/output206provides a microphone and speaker interface for use with the telephone facility. Universal Serial Bus (USB) input/output207is used to connect PDA511to another computer, or to the Internet110via a wired connection. GPRS/WiFi connection208and GSM connection209enable PDA511to connect to wireless networks, while Ethernet card210enables PDA511to connect to a wired network, for example via a docking station on a computer.

FIG. 3illustrates the contents of memory202of PDA104. An operating system301provides overall functionality for the device and content application302communicates with content server114to obtain and display data required by the user and to make transactions. Music player303is a plug-in that displays audio-visual data. This data may be stored on PDA104or provided by media server115.

Memory202also contains data303, which includes such data as undisplayed contents data or media data, packets to be sent or that have not been acknowledged, and data required by operating system301and content application302.

Media player303is a “black box” plug-in with a simple API. It will generally only play data that is sent using the Real Time Protocol (RTP), although it may also comprehend a proprietary protocol, and will accept a very limited number of commands. A media player application on, for example, a desktop computer would be much more sophisticated but since the invention described herein is designed to function on any kind of terminal it must interact with media players having the least amount of functionality, such as that described herein.

Audio-visual data is typically sent from an RTSP (Real Time Streaming Protocol) server to an RTSP client over UDP/IP using a communications link that is negotiated using RTSP over a TCP/IP link. It comprises data packets, sent in two RTP streams, one for video data and one for audio data, and two RTCP (Real Time Control Protocol) streams which contain control packets. A media player is generally configured to act as an RTSP client.

Steps performed by media player303are described inFIG. 4. At step401the player is initialized by operating system301or content application302. At step402a communications link is set up. This may be to data stored locally on flash memory203, as is likely if the player has been initialized by operating system301, or to a remote location as is the case if it has been initialized by content applications302. For the purposes of this description, the communications link is with media server115via Internet108.

At step403audio-visual data is received over the communications link and displayed on LCD204. Additionally, report packets are generated and sent over the communications link. These report packets confirm that the link is open and also contain information such as the number of lost packets.

At step404a STOP request is received, usually as a result of the user stopping playback on the device, and thus at step405a TEARDOWN request is sent over the communications link. This closes the communications link and at step406a reply is received confirming this. At step407a question is asked as to whether the user restarts playback and if this question is answered in the affirmative control is returned to step402and a new communications link is opened. If it is answered in the negative then the plug-in is terminated at step408.

FIG. 5details step402at which a communications link is defined by a series of Real Time Streaming Protocol (RTSP) messages sent over TCP. At step501a universal resource locator (URL) is received. This URL must start with “rtsp://” and be followed by an address of a server, a port number, and an identification of what is to be played, such as a file name. The URL is usually passed to the player in the form of a text string. At step502the player identifies the server address and port from the URL and at step503it makes a TCP connection with the identified port on the identified server. Once this TCP connection is made a DESCRIBE request is transmitted over it. This requests the server to describe the data that is going to be sent and thus at step505a reply is received and the required information is extracted at step506.

The information received includes an indication as to how many streams there are within the data. If the data is audio only then the indication will be that there is one stream, while for audio-visual data it will be two. Currently there is no other type of data that can be streamed but provision is made within the protocol for as many streams as are necessary. Thus at step507a SETUP request is transmitted for the first identified stream. This request includes the number of the port on the device to which the stream of data packets is to be delivered, and also the port to which the stream of control packets associated with the stream of data packets is to be delivered. Typically, these port numbers will be consecutive. At step508a reply is received from the server indicating the ports that it is using and at step509a PLAY request is sent to start the sending of packets.

At step510the question is asked as to whether there is another stream to be set up. If this question is answered in the affirmative then control is returned to step507and a SETUP request is transmitted again. However, if all streams have been set up then the question is answered in the negative and step402is completed.

Once this communications link is established then data is sent using RTP, typically over UDP since acknowledgements are not generally required but over TCP if UDP is not possible, to the two identified ports. Additionally, control packets are sent using the Real Time Control Protocol (RTCP) to the two additional ports. The original TCP connection may then be broken.

Thus it can be seen that for a typical communications link along which a stream of audio data and a stream of video data are to be sent, a total of six TCP requests must be made before data can be transmitted, and any number of these requests may need to be retransmitted if packets are lost. This can take a long time, particularly on a wireless link, and thus it may take between ten and twenty seconds to set up a communications link before data can be streamed to a terminal. This may be acceptable when the user wishes to view a single item, such as in a video conferencing environment. However, in the environment shown inFIG. 1the user of a terminal may be watching television channels on the terminal. Such a user typically wishes to change channel fairly frequently, for example when “channel surfing” to find a suitable programme to watch, when flicking between channels to check whether a programme has begun, when changing to another channel during a commercial break, and so on. Using the current system, every time the user wished to change channel a new communications link would have to be negotiated. Clearly, a user will not tolerate a twenty-second delay every time he wishes to change channel.

Once the communications link is set up, data packets are streamed to PDA104using RTP. For each RTP stream, there is an RTCP stream of control packets. Thus the data comprises two sets of streams, each set comprising a stream of data packets and a stream of control packets.

The control packets contain information necessary to allow media player303to display the audio-visual data correctly. In particular, they relate a display time to an server time. Each stream of RTP packets is considered to have a display time as exemplified in the graph shown inFIG. 6. This display time is linearly related to the server time and can be described using a gradient and an offset. The RTSP server that generates the RTP data packets also generates a random gradient and offset for each stream, (a preset gradient and random offset may be used instead). These are used to generate a display time for each RTP packet in that stream. The player receiving that packet must convert the display time into an server time in order to decide when the data included in that packet should be displayed.

As shown inFIG. 6, a video stream601includes a plurality of video data packets such as packets602,603, and604. An audio stream605includes a plurality of audio data packets such as packets606,607and608. These are more infrequent than the video data packets because audio data is typically smaller. Streams601and605have different gradients and different offsets. Thus at time t, the display time for video stream601is shown by line609while the display time for audio stream605is shown by line610. Thus two data packets that should be displayed at similar times may have very different display times.

Since the audio stream and video stream have different display times, additional information is required to display the audio and video data synchronously. Each control packet for each stream contains the server time at which the control packet was generated and the corresponding display time. Different players use this information in different ways, but typically the gradient is defined during setup, and an extrapolation is made from the last few received control packets to define an adder which when combined with the defined gradient gives a time definition. This is used to determine what the server time is of a received RTP packet. Alternatively, a client may also be able to determine the gradient using extrapolation.

FIG. 7details step404at which data is received and displayed by media player303. At step701a packet is received and at step702a question is asked as to whether it is a control packet, which can be determined by which port a packet arrives on. If this question is answered in the affirmative then at step703the time definition for the relevant stream is updated. If the question is answered in the negative then the packet is a data packet and at step704the display time for the data packet is calculated from the current time definition. At step705the packet is displayed at the correct time. It may be that data packets received later than the current data packet should be displayed earlier, or vice versa. This is because packets may be routed differently and take a shorter or longer time to arrive. Also, packets may be lost and thus there may be times at which no data is displayed at all. However, this is not likely to be very noticeable to the user as long as the packet loss is not too high.

At step706a question is asked as to whether a report packet should be generated. These are sent every few seconds, and thus if the question is answered in the affirmative then a report packet is generated and transmitted at step707. Amongst other information, this report packet contains a loss fraction. Because all data packets are numbered sequentially it is possible for the player to know how many packets have not been received and this information is included in the report packet.

At this stage, and following step703, a question is asked as to whether another packet has been received. If this question is answered in the affirmative then control is returned to step702and the packet is processed. If it is answered in the negative then step404is completed.

An illustration of PDA104is shown inFIG. 8. Audio-visual data is displayed on LCD204and the user can control the device using buttons801. In the illustration, the user is viewing a news channel.

According to prior art systems, if the user wished to change channel the player303would terminate the communications link and negotiate a new one as described with respect toFIG. 5, which could take as long as twenty seconds. However, according to the invention described herein, the user may change channel much more quickly.

FIG. 9details content application302. This is an application loaded on PDA104that communicates with content server114via internet108in order to supply content to the user as required and also to facilitate transactions made by the user. Content application302communicates with media player303using the limited range of commands that the player will accept, typically only PLAY, STOP or PAUSE. It will also accept a text string containing a URL as described with reference toFIG. 5. At step901the content application302initializes when the PDA104is switched on. At step902the application obtains and displays content as required by the user. At step903a question is asked as to whether the user wishes to view media, indicated by the user making certain keypresses using keys801based on options displayed on LCD204, and if this question is answered in the affirmative then at step904audio-visual data is obtained and displayed. At step905a question is asked as to whether the user is closing the application. This usually occurs when the device is switched off, but the application can be closed at any time. If this question is answered in the negative then control is returned to step902and if it is answered in the negative then the application closes at step906.

FIG. 10details step904during which audio-visual data is obtained and displayed. At step1001a channel guide is obtained and displayed on LCD204. This guide indicates to the user which programs are being shown on available channels and at what time. Channels that are not available to the user may also be shown, for example to encourage the user to increase his subscription level. At step1092the user selects a channel to view and at step1003a CHANNEL PLAY request is sent to content server114. The content server generates a URL and returns it to the device over internet108and thus at step1004a question is asked as to whether a URL has been received. If this question is answered in the negative then either no message or an error message has been received from content server114, and so the message “CHANNEL NOT PERMITTED” is displayed at step1005before control is returned to step1002for a new channel selection. However, if it is answered in the affirmative then at step1006media player1003is initialized and at step1007the received URL is passed to the player in the form of a string of text. The URL points the player to media server115, and upon receipt the player carries out the steps shown inFIG. 5to set up a communications link to start streaming and displaying audio-visual data.

At step1008a question is asked as to whether the user wishes to change the channel, indicated by the user making certain keypresses using keys801based on options displayed on LCD204. If this question is answered in the affirmative then at step1009a CHANNEL CHANGE request is sent to content server114. As will be described further with respect toFIG. 14, upon receipt of this request content server114requests media server115to change the data sent via the established communications link. Thus the player receives different data, ie a new channel, without having to terminate and renegotiate the communications link. The channel change thus appears virtually seamless to the user.

Thus at step1010a question is asked as to whether the message “NOT PERMITTED” has been received, and if this question is answered in the affirmative then the message “CHANNEL NOT PERMITTED” is displayed to the user. Otherwise, the channel change has been carried out and in both cases then control is returned to step1008. Eventually the question asked at step1008is answered in the negative and at step1012the user stops play in the media player. The player is closed at step1013and step904is complete.

PDA104is again illustrated inFIG. 11. The user has indicated that he wishes to change channel and so content application302overlays a channel guide1101on the display204. The user may select a channel to watch using buttons801, following which the content application302contacts the media server114to change channel. Media server115sends the packets of a different channel down the existing communications link and thus the channel change is transparent to the media player303. However, because each data stream of the new channel will have a different display time from the current streams, they cannot be simply switched over. The player will continue to apply its existing time definitions to the new packets, thus leading to a calculation of server time that could be wrong by seconds, minutes, days or even years. Further, the new channel's data streams will have different packet sequence numbers from the original streams.

FIG. 12shows content server114. It comprises two parallel central processing units (CPUs)1201and1202having a clock frequency of 3 GHz, a main memory1203comprising 4 GB of dynamic RAM and local storage1204provided by a 20 Gb-disk array. A CD-ROM disk drive1205allows instructions to be loaded onto local storage1204from a CD-ROM1206. A first Gigabit Ethernet card1207facilitates intranet connection, and can also be used for installation of instructions. A second Gigabit Ethernet card1208provides a connection to Internet108.

The contents of main memory1203are illustrated inFIG. 12. Operating system1301provides operating system instructions for common system tasks and device abstraction. In this example a Windows® Server operating system is used, but another system providing similar functionality could be used. Content serving applications1302include instructions for delivering content to terminals, updating personal details, making transactions, and so on. Media controller1303receives requests from terminals for the playing of audio-visual data, including channel change requests, and communicates with media server115. Data1304includes session data for each user, buffered messages, and other data used by operating system1301, content serving application1302and media controller1303.

FIG. 14details steps carried out by media controller303to serve requests from terminals such as PDA104. At step1401it starts, typically during the starting of content server114, and at step1402a request to view a channel is received from a content application on a terminal, such as that sent by content application302at step1003. At step1403a question is asked as to whether the user is permitted to view the requested channel. This is done by loading user data from hard drive1204into main memory1203and checking the user permissions. If the question is answered in the negative then at step1404the message “NOT PERMITTED” is sent back to the requesting terminal at step1404. However, if it is answered in the affirmative then at step1405a question is asked as to whether the request is a CHANNEL PLAY request, indicating that a new communications link needs to be opened, or a CHANNEL CHANGE request, indicating that the link is open but that a new channel is required.

If the answer is CHANGE then at step1406a CHANNEL CHANGE request is sent in turn to media server115. This request identifies the requesting device and the requested channel. At step1407a further question is asked as to whether a reply of “OK” is received. If this question is answered in the negative then for some reason the media server cannot change channel, probably because the communications link has been broken. Thus at this stage, or if the received request is a CHANNEL PLAY request, a request for a new URL is sent to media server115. At step1409the URL is received and at step1410it is sent to the requesting terminal in order that the terminal can open a TCP connection using the URL. The media controller takes no further part in the set-up of the communications link.

At this stage, or following an “OK” reply at step1407, a question is asked at step1411as to whether another request has been received. If this question is answered in the affirmative then control is returned to step1403and the process is repeated. Eventually the question is answered in the negative and the process is shut down at step1412, usually with the switching off for some reason of content server114.

Thus media controller acts as an intermediary between a terminal such as PDA104and media server115, checking that a user is permitted to view channels before requesting media server115to fulfill the request.

FIG. 15shows media server115, which is substantially similar to content server114. It comprises two parallel central processing units (CPUs)1501and1502having a clock frequency of 3 GHz, a main memory1503comprising 4 GB of dynamic RAM and local storage1504provided by a 20 Gb-disk array. A CD-ROM disk drive1505allows instructions to be loaded onto local storage1504from a CD-ROM1506. A first Gigabit Ethernet card1507facilitates intranet connection to RTSP servers116,117and118. A second Gigabit Ethernet card1508provides a connection to Internet108.

Media server115receives data streams from RTSP servers116to118that are forwarded to the terminals on request. Media server115sets up a plurality of server channels1601,1602,1603,1604,1605,1606,1607,1608and1609, each of which emulates an RTSP client and negotiates a communications link in the usual way in order to receive audio-visual data from the RTSP servers. Thus, for example, server channel1601negotiates communications link1610with RTSP server116in order to receive the two RTP streams and two RTCP streams that define a first television channel. This audio-visual data contains the same programmes that are sent over the usual television and satellite networks, but it is encoded suitably for display on terminals, typically reducing the amount of data considerably. In this embodiment all the data is encoded using the same encoder to allow easier switching of channels, but in other embodiments transcoding of the outgoing data could be used at the server, thus allowing different encodings of the different incoming channels.

For each terminal that has requested audio-visual data a user channel is defined, such as user channel1611that communicates with PDA104, user channel1612that communicates with PC106, user channel1613that communicates with mobile telephone103, and user channel1614that communicates with mobile telephone102. Each user channel emulates an RTSP server in order to communicate with the media player on its respective terminal.

Each user channel receives input from one server channel. Thus, for example, user channel1611receives input from server channel1601, user channels1612and1613both receive input from server channel1604, and user channel1614receives input from server channel1608. These inputs are the data received from the RTSP servers by the respective server channel. On first set-up of a user channel the input data is passed by the user channel to the terminal without alteration. However, upon fulfillment of a channel change request the input is changed. Thus, when PDA104requests a change of channel to the channel provided by server channel1603the media server115changes the input to user channel1611. The input represented by line1615is stopped and instead the data from server channel1603is input, as represented by line1616.

However, once the channel has been changed the data can no longer be sent unaltered to PDA104. The display time within each RTP and RTCP packet must be altered before being sent. Further, since the media player is expecting packet numbers to continue in sequence, the sequence numbers of the packets must also be altered.

The contents of main memory1503are illustrated inFIG. 17. Operating system1701provides operating system instructions for common system tasks and device abstraction. In this example a Windows® Server operating system is used, but another system providing similar functionality could be used. Channel manager1702receives requests from media controller1303and manages the server channels and user channels. It also includes a Quality of Service manager1703that monitors report packets received from terminals. Server channel objects1704define server channels1601to1609while user channel objects1705define user channels1611to1614. Data1706includes data used by operating system1701and channel manager1702.

FIG. 18details steps carried out by channel manager1702. At step1801it starts up, usually with the switching on of media server115. At step1802the first connected RTSP server is selected and at step1803the first channel that the server supplies is selected. At step1804a server channel object is defined to receive the data and at step1805a communications link is defined between the RTSP server and the server channel as an RTSP client. At step1806a question is asked as to whether there is another channel on the server, and if this question is answered in the affirmative then control is returned to step1803and the next channel is selected. If it is answered in the negative then at step1807a further question is asked as to whether there is another connected RTSP server, and if this question is answered in the affirmative then control is returned to step1802and the next server is selected. Alternatively, if the question is answered in the negative then all the necessary server channels have been defined.

Thus at step1808a request to play a channel is received from media controller1303on content server114. At step1809a question is asked as to whether the request is a CHANNEL PLAY or a CHANNEL CHANGE request. If the answer is PLAY then at step1810a new user channel is defined. Alternatively, if it is a CHANGE request then at step1811the existing user channel for the requesting terminal is modified. Following either step, a question is asked at step1812as to whether another request has been received, and if this question is answered in the affirmative then control is returned to step1809and the request is processed. Eventually the question is answered in the negative and channel manager1702is shut down at step1813, usually with the switching off for some reason of media server115.

FIG. 19details step1810at which a new user channel is defined. At step1901a question is asked as to whether a user channel already exists for the requesting terminal. This may happen when a communications link is broken and the terminal needs to renegotiate the link with a new CHANNEL PLAY request, and so if the question is answered in the affirmative then the existing user channel is deleted at step1902. At this stage, or if the question is answered in the negative, then a user channel is created at step1903and at step1904a URL is created that indicates either the IP address or a resolvable DNS address of the media server, the TCP port that has been allocated to the user channel, and a filename that will be recognized by the channel manager as indicating the defined user channel. This URL is then returned to the media controller1303at step1905in order that it can be sent in turn to the requesting terminal. The URL is time limited for security purposes and thus at step1906a question is asked as to whether the terminal used the URL to open a TCP connection within thirty seconds and if this question is answered in the negative then at step1907the user channel is deleted. Alternatively, the TCP connection is initiated in time and the user channel is set up at step1908.

FIG. 20details step1908at which the user channel is set up, mainly by establishing a communications link between the requesting terminal as an RTSP client and the defined user channel as an RTSP server. On the terminal side this is performed by the media player, such as media player303, carrying out the steps detailed inFIG. 5. Thus at step2001the channel manager1702receives a DESCRIBE request from the terminal and at step1703it sends a reply that includes an indication of the number and type of streams, such as one audio stream and one video stream. At step2003a SETUP request is received from the terminal for the first stream, which includes the terminal port numbers that should be used for the first RTP and RTCP data streams. A reply is sent at step2004indicating the ports on the media server that the user channel is using for these streams. At step2005a question is asked as to whether there is another stream, and if this question is answered in the affirmative then control is returned to step2003and a SETUP request is received for that stream. Alternatively, the question is answered in the negative and the TCP connection is closed at step2006. At step2007the necessary ports are opened and at step2008initial user channel offsets of zero are stored in the user channel object. This means that the display times contained in the data packets will be offset by zero, ie left unaltered, as will the sequence numbers of the packets.

Step1811at which a user channel is altered following a CHANNEL CHANGE request is detailed inFIG. 21. At step2101a question is asked as to whether the user channel is still open, since it is possible that the channel manager may have closed it due to a long period of inactivity following a broken communications links. If this question is answered in the affirmative then at step2102the message “NOT OK” is sent back to media controller1303and step1811is exited. However, under normal circumstances the user channel is open and sending data to the terminal. Thus at step2103the channel offsets, ie the display time offsets and the sequence number offsets, are changed in the user channel object and at step2104the input to the user channel is changed to the server channel that has the requested television channel. The message “OK” is then sent to media controller1303at step2105.

FIG. 22illustrates the effect of changing the display time offsets at step2103, which will be described further with respect toFIG. 23. The graph plots server time against display time for four RTP streams, a first video stream2201and a first audio stream2202, which are the streams that are the current input into the user channel, and a second video stream2202and a second audio stream2204, which carry the television channel that the user wishes to change to.

For each of the streams, the display time corresponding to an server time T shown by line2205can be calculated from the RTCP packets corresponding to each stream. This is done in this example by extrapolating from the last four RTCP packets for each stream. The difference in display times D1, shown by arrow2206, between the two video streams and the difference in display times D2, shown by arrow2207, between the two video streams can then be calculated. D1is added to the current video offset and D2is added to the current audio offset, both of which are zero at first set-up of the user channel, to produce a video offset2208and an audio offset2209. Every packet that is sent by the user channel is altered by adding the video offset to the display time in the RTP and RTCP packets for the video stream, and by adding the audio offset to the display time in the RTP and RTCP packets for the audio stream.

Thus the second video stream is offset as shown by line2210and the second audio stream is offset as shown by line2211. The player receiving the packets will thus display them at a modified server time. The gradients of the streams are different, leading to a slight speeding up or slowing down at first, but this will be corrected once two or three RTCP packets have been received by the player. The user will not notice this slight change in speed as long as the audio and video streams are synchronized.

In an embodiment where the client is not able to change the gradient once it has been set, all the streams would have to have the same gradient. In practice, it is likely that servers116,117and118will use preset gradients rather than randomly-generated ones and thus all the streams would have the same gradient.

The internal clocks on servers116,117and118may not be synchronized, and if this is the case then the server time in data packets originating from different servers will be different. The result of this is that when switching between streams from different servers there may be either a jump backwards or a delay in viewing of the data, which will be equal to the difference between the servers' internal clocks. A jump backwards in time would probably not be noticed unless the server time were extremely inaccurate, but a delay would be noticed and not tolerated by a user. Thus the offsets may need to be augmented to take account of this fact, and this is described further with reference toFIG. 23a.

FIG. 23details step2103at which the channel offsets are changed as illustrated inFIG. 22. At step2301an server time T is defined and at step2302a first type of stream, video or audio, is selected. At step2303the display time of the stream that is no longer required is calculated for the server time T, and at step2304the same is calculated for the new stream. At step2305the difference D between the display times is calculated and at step2306this difference is added to the current display time offset for the selected stream type. At step2307the packet sequence number difference is calculated by subtracting the sequence number of the next packet in the old stream from the sequence number of the next packet in the new stream and adding it to the current sequence number offset for the selected stream type at step2308.

At step2309a question is asked as to whether there is another type of stream and if this question is answered in the affirmative then control is returned to step2302and the next type of stream is selected. Alternatively, the question is answered in the negative and at step2110both of the display time offsets are augmented.

The display time offsets for both the video and audio streams may need to be augmented to take account of different server times. If the old and new channels come from the same server, or servers having synchronized internal clocks, this step will result in no change to the offsets. However, if the channels come from different servers having non-synchronized clocks this step will ensure a smooth transition between channels.

At step2311the video stream in the new channel is selected and at step2312the calculated display time offset for the video stream is added to it. At step2313the last packet that was sent in the old video data stream is selected and at step2314the difference between the display time in this last packet and the offset display time in the new packet is calculated. Between two channels with no difference in server time, this difference will be zero, but between channels coming from different, non-synchronized servers, this difference will not be zero.

Thus at step2315the difference is converted to a server time using the function for the old video stream, and at step2316this server time is converted to display time using the function for the new video stream. The result of this is added to the video display time offset at step2317. Similarly, at step2318the same server time is converted to display time using the function for the new audio stream. The result of this is added to the audio display time offset at step2319.

This means that for both audio and video, the first data packet in the new stream has the same display time as the last packet in the old stream and thus any difference between server clocks is allowed for.

FIG. 24shows a further illustration of the changing of display time offsets (assuming that the servers are synchronized and no augmentation is necessary). The input to the user channel has changed to the second video stream2203and2204, but modified data packets are being sent with an offset display time as shown by lines2210and2211. Another CHANNEL CHANGE request has been received and so the input to the user channel will change to third video stream2401and third audio stream2402. Thus at time T, shown by line2403, the display times for the modified second video and audio streams2210and2211and the third video and audio streams2401and2402are calculated. The difference in display times D1, shown by arrow2404, between the two video streams and the difference in display times D2, shown by arrow2405, between the two audio streams can then be calculated. D1is added to the current video offset and D2is added to the current audio offset to produce a video offset2406and an audio offset2407. Thus the third video stream is offset as shown by line2408and the second audio stream is offset as shown by line2409.

FIG. 25is a block diagram showing the streams that pass along the communication links. Media server115communicates with a terminal, for example PDA104, using user channel1611which is taking input from, for example, server channel1608. RTP data packets2501are received by server channel1608from RTSP server116and passed to user channel611, which applies the display time and sequence number offsets and sends them to PDA104along communications link2502as modified RTP data packets2503. RTCP control packets2504are received by server channel1608and passed to user channel611, which applies the offsets and sends them to PDA104along communications link2502as modified RTCP data packets2505.

Server channel1608creates RTCP report packets2506and sends them to the RTSP server116. PDA104also creates report packets2507and sends them via communications link2502to user channel1611, which forwards them to channel manager1702.

The bandwidth available to communications links over internet108fluctuates constantly and thus at any one time the packet loss may be very high or very low. RTCP report packets include an indication of packet loss, expressed as a fraction known as a loss fraction, and Quality of Service (QoS) manager1703, which is part of channel manager1702, uses this information to alter the service supplied to PDA104. If the packet loss is high then there will be many gaps in the audio-visual data being viewed by the user, particularly if keyframes are being lost. The user will probably prefer to view a lower quality version of the television channel than continue with gaps in the viewing. Thus when packet loss is very high a downwards adjustment in quality can be made.

One way in which this can be performed is by having more than one encoding of each television channel, with each encoding being of a different quality, and thus having a different amount of data. Thus, for example, referring back toFIG. 16, server channel1601receives data representing a first television channel encoded at 30 kbits/second, server channel1602receives the same television channel but encoded at 50 kbits/second, and server channel1603receives the same television channel but encoded at 100 kbits/second. Thus each server channel receives audio-visual data representing the same information, ie the same television channel, with each audio-visual data being encoded at a different data rate. Because the method of channel switching described herein provides virtually seamless switching, it is possible to step down, ie switch from a higher encoding to a lower encoding, transparently to the user. Thus first audio-visual data containing first information encoded at a first data rate and second audio-visual data containing first information encoded at a second data rate that is less than the first data rate are received. The first audio-visual data is provided to a network-connected terminal, before a condition to the effect that the amount of data being sent to the terminal exceeds the available bandwidth is identified. Provision of the first audio-visual data to the terminal is stopped and the second audio-visual data is provided instead.

However, because of the nature of the Internet108it is extremely likely that temporary high packet loss will occur occasionally, leading to a step down in quality. It is therefore necessary to step up, ie switch from a lower encoding to a higher encoding, should packet loss be very low. Thus a condition to the effect that the amount of data being sent to the terminal is less than the available bandwidth is identified, provision of the second audio-visual data to the terminal is stopped, and the first audio-visual data is provided again to the terminal.

However, low packet loss only indicates that the current amount of data is acceptable. For example, if data is being sent at 50 kbits/second and there is 70 kbits/second of available bandwidth then packet loss will be low, but attempting to increase the data to 100 kbits/second will result in extreme packet loss and, contrary to intentions, a degradation of quality for the user. Thus instead of stepping up immediately, QoS manager1703sends probe packets2508, which contain dummy data, to the same ports on the receiving terminal as the RTCP packets. They are discarded by the media player because they are not RTCP packets. By increasing the number of probe packets2508sent until the total amount of data being transmitted is equal to the amount required by the higher encoding, QoS manager1703can ensure that the bandwidth required is available before stepping up. Thus the difference between the data rates of the first and second audio-visual data is calculated, a condition to the effect that the difference is larger than a specified threshold is identified, and additional data packets are sent to the terminal to determine whether the available bandwidth is large enough to provide the first audio-visual data to the terminal.

FIG. 26shows the steps carried out by QoS manager1703. At step2601it starts up when channel manager1702starts and at step2602a report packet is received from a user channel. At step2603the corresponding user channel object is selected and at step2604the data in the object is examined to determine whether more than two report packets have been received since the last logged event. An event may be a step down in quality, a step up, or a failed attempt to step up. The loss fraction in the first two report packets after an event tends to be distorted by the event and so they are not considered. Thus if the question asked at step2604is answered in the affirmative then at step2605the loss fraction contained in the report packet is stored in the user channel object.

Loss fractions are examined in samples of four and so at step2606a question is asked as to whether there are four loss fractions stored in the user channel object, and if this question is answered in the affirmative then a further question is asked as to whether two of the loss fractions are greater than 100/256. If this question is answered in the affirmative then at step2608a failed sample is logged, whereas if it is answered in the negative a passed sample is logged at step2609. In either case, the loss fractions are cleared from the object in order to start a new sample.

The quality of the channel is then changed if appropriate at step2611and at step2612a question is asked as to whether there is another report packet. This step also follows an answer in the negative to the question asked at step2604or step2606. If it is answered in the affirmative then control is returned to step2603. Eventually it is answered in the negative and the manager terminates at step2613, usually with the switching off for some reason of media server115.

FIG. 27details step2611at which the quality of the channel is changed if appropriate. At step2701a question is asked as to whether a period of time, in this example at least thirty seconds, has lapsed since the last event, again because events can distort the data. If this question is answered in the affirmative then at step2702a question is asked as to whether there have been three consecutive passed samples. If this question is answered in the affirmative then a step up in quality may be appropriate and this is carried out at step2703. Alternatively, if the question asked at step2702if this question is answered in the negative then a further question is asked at step2704as to whether the last sample failed. If this question is answered in the affirmative then a step down in quality may be appropriate and this is carried out at step2705. If this question or the question asked at step2701is answered in the negative then no change in quality is made.

FIG. 28details step2703at which a step-up in quality may be carried out. If there have been three consecutive passed samples then clearly the current amount of data being sent on the communications link is getting through and there may be bandwidth for more data, and thus a higher quality of picture, both audio and video, could be shown. If the jump in the amount of data between encodings is not very large then an attempt at stepping up could be made, since the likelihood of stepping far beyond the packet rate that can be supported is low, and even if the new rate cannot be supported a step back down can be made without having degraded the user experience significantly. Alternatively, probe packets are sent to check the available bandwidth.

Thus at step2801a question is asked as to whether probe packets are currently being sent. If this question is answered in the negative then at step2802a question is asked as to whether a higher-quality encoding of the television channel is available. If this question is answered in the negative then there is no need to pursue the question of stepping up and so step2703is terminated. However, if it is answered in the affirmative then at step2803a further question is asked as to whether the difference in bandwidth between the current encoding and the higher-quality encoding is greater than 32 kbits/second. If this question is answered in the affirmative then the sending of probe packets2708is commenced at step2804, as will be described further with reference toFIG. 30. Alternatively, if the question is answered in the negative then the difference between encodings is small enough to attempt a speculative step-up and thus at step2805a CHANNEL CHANGE request for the higher-quality encoding is sent to channel manager1702, which processes it in the same way as any other CHANNEL CHANGE request.

FIG. 29details step2705at which a step down in quality may be made. At step2901a question is asked as to whether probe packets are currently being sent. If this question is answered in the affirmative then the failed sample means that a step up will not be possible and thus at step2902the probe packets are stopped by terminating the process described with reference toFIG. 30. However, if it is answered in the negative then at step2903a further question is asked as to whether a lower-quality encoding of the television channel is available. If this question is answered in the negative then nothing can be done about the packet loss and step2705is terminated. However, if it is answered in the affirmative then at step2904a CHANNEL CHANGE request for the lower-quality encoding is sent to channel manager1702, which processes it in the same way as any other CHANNEL CHANGE request.

Thus a failed sample results in the cessation of probe packets if they are being sent, or in a step down in quality if they are not being sent

FIG. 30illustrates the probing mechanism, which is a separate thread on QoS manager, and which is started at step2804after three consecutive passed samples when the difference in bandwidth between versions is greater than 32 kbits/second. Thus at step3001the mechanism is started. At step3002a step size is calculated to be one tenth of the difference in bandwidth between the versions, and at step3003a probing interval is calculated as eight times the average time between receipt of report packets. At step3004the probing starts by sending dummy data that increases the bandwidth used by a number of step sizes, called the level. Probing starts at level one.

The process then waits at step3005for the calculated probing interval before asking the question at step3006as to whether level ten of probing has been reached yet. If this question is answered in the negative then the level is increased by one at step3007and control is returned to step3005where the process again waits. Eventually the question asked at step3006is answered in the affirmative, to the effect that level ten has been reached. This means that the amount of bandwidth required by the probe data plus the current version is equal to the amount of bandwidth required by the higher quality version. The process waits at step3008for twice the probing interval to ensure that this bandwidth is sustainable before sending a CHANNEL CHANGE request for the higher-quality encoding to channel manager1702at step3009, which processes it in the same way as any other CHANNEL CHANGE request. The process then terminates at step3010.

Alternatively, should a failed sample be logged at any time then the QoS manager will terminate the probing process at step2902.

FIG. 31illustrates how the QoS manager1703responds to changes in available bandwidth, indicated by the packet loss fraction, by stepping up or stepping down the quality of the television channel supplied to a terminal. The line3101plots available bandwidth in kbits/second against time. Four encodings of the same channel are available and are shown on the bandwidth axis according to how much bandwidth they require. The first encoding is at 30 kbits/second, the second at 50 kbits/second, the third at 100 kbits/second and the fourth at 160 kbits/second.

At first, as shown by line3102, the third encoding at 100 kbits/second is transmitted. The available bandwidth3101starts to fall, and indeed at point3103falls below 100 kbits/second but not for long enough to produce a failed sample. However, at time3104a failed sample leads to a step-down in quality to the second encoding at 50 kbits/second. As the bandwidth3101continues to fall a further step-down is made at time3105to the lowest-quality encoding at 30 kbits/second.

The bandwidth starts to improve and three consecutive passed samples at time3106lead to a step-up in quality to 50 kbits/second. Since the difference between the encodings is only 20 kbits/second a speculative step-up rather than probe data is possible. At time3107probing starts, and thus while the amount of RTP/RTCP data being sent is still at 50 kbits/second, as shown by line3108, the total data sent over the communications link is higher, as shown by line3109. However, the available bandwidth3101is not high enough and thus at time3110a failed sample means that the probing is stopped. Probing starts again at time3111and this time is successful, resulting in a step up to the third encoding at 100 kbits/second at time3112. The bandwidth continues to rise and thus at time3113probing starts again in order to test the possibility of a step up to the highest encoding.