Repository: postlund/pyatv
Pull request #2402 (2024-06-18 10:32:40)

Added touch gesture to companion protocol
Hi, I have added the implementation of touch gestures and updated the documentation.

I have tested it successfully.
To test it :
```python
atv = await pyatv.connect(config, loop)
companion = cast(FacadeRemoteControl, atv.remote_control).get(Protocol.Companion)
if companion:
    api: CompanionAPI = companion.api
    # Simulate a gesture from x=100, y=100 to x=1000, y=1000 in 1000 milliseconds
    await api.touch_gesture(100, 100, 1000, 1000, 1000)
```

albaintor: Hi, I have added the new interface and its implementation including the facadeAppleTV class
IsakTheHacker: Hi, amazing work you've done here! Have been waiting for this for so long. Do you think it would be possible to have this working for real time gestures, just like the way it works for the Siri Remote and in control center on iOS? I have no idea how such an interface would work or even look like though, but that would be very awesome. Everything looks exceptionally good by the way and I hope @postlund can take a look at this soon
albaintor: 
https://github.com/postlund/pyatv/assets/118518828/a3bd72ce-8726-42b6-a017-05d64fac83fa

I have made a tool to simulate the touchpad gestures and the right algorithm behind
postlund: A section in the `atvremote` documentation with some examples would be great as well!
postlund: One last thing that I forgot about... Some kind of basic tests for the added API methods are needed. Only a "good case" is needed, just to ensure we verify something. Should be fairly easy to add in https://github.com/postlund/pyatv/blob/master/tests/protocols/companion/test_companion_functional.py
albaintor: > A section in the `atvremote` documentation with some examples would be great as well!

Yes I will add that
albaintor: > One last thing that I forgot about... Some kind of basic tests for the added API methods are needed. Only a "good case" is needed, just to ensure we verify something. Should be fairly easy to add in https://github.com/postlund/pyatv/blob/master/tests/protocols/companion/test_companion_functional.py

I don't think this is possible : touch events are sent with no feedback by Appletv
postlund: > > One last thing that I forgot about... Some kind of basic tests for the added API methods are needed. Only a "good case" is needed, just to ensure we verify something. Should be fairly easy to add in https://github.com/postlund/pyatv/blob/master/tests/protocols/companion/test_companion_functional.py
> 
> I don't think this is possible : touch events are sent with no feedback by Appletv

I have implanted fake "server sides" of each protocol. So you can just store the events and verify what you received. No interaction with an Apple TV is used (it's just unit tests).
albaintor: > > > One last thing that I forgot about... Some kind of basic tests for the added API methods are needed. Only a "good case" is needed, just to ensure we verify something. Should be fairly easy to add in https://github.com/postlund/pyatv/blob/master/tests/protocols/companion/test_companion_functional.py
> > 
> > 
> > I don't think this is possible : touch events are sent with no feedback by Appletv
> 
> I have implanted fake "server sides" of each protocol. So you can just store the events and verify what you received. No interaction with an Apple TV is used (it's just unit tests).

Ok got it, I am on it
albaintor: Hi,
all done. I'll let you review. I have also updated the atvremote script to support touch calls
albaintor: As an example :
```shell
# Horizontal touch gesture from left to right in 200ms
atvremote.py --id <appletvid> touch=0,0,800,0,200
# Touch click
atvremote.py --id <appletvid> touch_click
```
postlund: I'll try to look at it tonight, great work so far! Also remember that we should add some developer documentation as well at some point, in case someone wants to use it (a subpage here https://pyatv.dev/development/).
albaintor: > I'll try to look at it tonight, great work so far! Also remember that we should add some developer documentation as well at some point, in case someone wants to use it (a subpage here https://pyatv.dev/development/).

Yes I a little new to python's coding (since this year), however I found your code very well organized apart all the offered features for all-in-one library.
I got a little struggled though by the fact that I had to apply modifications in several files (don't remember exactly but : companion/api, companion/protocol, interfaces, and facadeappletv)
thiccaxe: I'd also ask you to take a look at "press and hold"
This is the same as regular click, except:
press:
```js
{'_i': '_hidC', '_x': 4067597582, '_btHP': False, '_c': {'_hBtS': 1, '_hidC': 6}, '_t': 2}
```
wait, say 1 second or something, then:
```js
{'_i': '_hidT', '_x': 4067597583, '_c': {'_ns': 86787906162458, '_tFg': 1, '_cx': 1000, '_tPh': 5, '_cy': 1000}, '_t': 1}
{'_i': '_hidC', '_x': 4067597584, '_btHP': False, '_c': {'_hBtS': 2, '_hidC': 6}, '_t': 2}
```

albaintor: > I'd also ask you to take a look at "press and hold" This is the same as regular click, except: press:
> 
> ```js
> {'_i': '_hidC', '_x': 4067597582, '_btHP': False, '_c': {'_hBtS': 1, '_hidC': 6}, '_t': 2}
> ```
> 
> wait, say 1 second or something, then:
> 
> ```js
> {'_i': '_hidT', '_x': 4067597583, '_c': {'_ns': 86787906162458, '_tFg': 1, '_cx': 1000, '_tPh': 5, '_cy': 1000}, '_t': 1}
> {'_i': '_hidC', '_x': 4067597584, '_btHP': False, '_c': {'_hBtS': 2, '_hidC': 6}, '_t': 2}
> ```

I was wondering if I had covered all the touch gestures and I missed it... to be done indeed
albaintor: > I'll try to look at it tonight, great work so far! Also remember that we should add some developer documentation as well at some point, in case someone wants to use it (a subpage here https://pyatv.dev/development/).

Hi,
I have fixed a bug in touch click (delay needed between commands otherwise it won't work all the time) and added the missing touch_hold command. Please let me know if there is something  more to be done.

Also Pierre I am interested in adding voice commands to the library (send siri commands in audio or text format). Your proxy program could sniff those requests too ? Thanks
postlund: Great, I will take a look! Trying to catch up without work ask I can make release with this soon too.

I have not tested it, but I expect that it should work. I guess you will have to test. One thing to keep in min though is that we should try to re-use as much work as possible from AirPlay streaming when providing audio/voice. I expect that the user should be able to pick synthesizer and just provide a text string to speak or just provide a pre-recorded file. We need to be flexible here.
albaintor: Hi Pierre,

There is a regression on your recent changes : the modifications applied on support/chacha20.py broke the support of companion protocol
https://github.com/postlund/pyatv/commit/d5a29d7efbbaf05920b08a4e54bea178ddf883b2

If I revert those changes it works again

Thanks

De : Pierre Ståhl ***@***.***>
Envoyé : samedi 22 juin 2024 10:46
À : postlund/pyatv ***@***.***>
Cc : albaintor ***@***.***>; Author ***@***.***>
Objet : Re: [postlund/pyatv] Added touch gesture to companion protocol (PR #2402)


Great, I will take a look! Trying to catch up without work ask I can make release with this soon too.

I have not tested it, but I expect that it should work. I guess you will have to test. One thing to keep in min though is that we should try to re-use as much work as possible from AirPlay streaming when providing audio/voice. I expect that the user should be able to pick synthesizer and just provide a text string to speak or just provide a pre-recorded file. We need to be flexible here.

-
Reply to this email directly, view it on GitHub<https://github.com/postlund/pyatv/pull/2402#issuecomment-2183945418>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A4IHILECACAR4DCX6QQPSRDZIU2WBAVCNFSM6AAAAABJPXY7ZSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBTHE2DKNBRHA>.
You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

postlund: I see the regression as well and I know why, will fix it later.

You can trigger Siri by holding the power button with the remote up (like you do to trigger Siri in iOS) I think.
thiccaxe: Voice is easy, the hard part is figuring out the format apple uses. The best option for now might be to capture packets from an I-device and then when requested re-send those with pyatv. Atleast as an initial implementation 
thiccaxe: https://github.com/postlund/pyatv/issues/2325#issuecomment-2001059905
thiccaxe: AFAIK you need Siri enabled on the Apple TV AND iPhone for it to work...
postlund: I'm pretty sure I did dumps of some voice interaction back in the days over MRP. Must have misplaced the protobufs I made for that, as I'm pretty sure there were mentions of things such as format (e.g. PCM8/16/24) and actual audio samples. Would expect something similar would be used here as well.
albaintor: I have committed changes, you can review them. 
albaintor: > [#2325 (comment)](https://github.com/postlund/pyatv/issues/2325#issuecomment-2001059905)

Hi, I have managed to make siri commands work using text but only if I enable "write to siri" in accessibility options : this raises a text field that I can play with. Otherwise it expects a voice command. I didn't find out how to provide text commands with another method (is it possible ?)
Here is the sample code in the companion API : for example if you call _test_siri("time") it will display current time on AppleTV
Otherwise your link is not enough to digg into voice commands : I know how to raise a siri start/stop command, but the tough part is the requests with payloads (audio protocol...)


```python
    async def _test_siri(self, command):
        await self._siri_start()
        await asyncio.sleep(1)
        await self._siri_stop()
        await self.text_input_command("", clear_previous_input=True)
        await asyncio.sleep(0.1)
        await self.text_input_command(command, clear_previous_input=True)
        await self.hid_command(True, HidCommand.Down)
        await self.hid_command(False, HidCommand.Down)
        await asyncio.sleep(0.1)
        await self.hid_command(True, HidCommand.Down)
        await self.hid_command(False, HidCommand.Down)
        await asyncio.sleep(0.1)
        await self.hid_command(True, HidCommand.Down)
        await self.hid_command(False, HidCommand.Down)
        await asyncio.sleep(0.1)
        await self.touch_click(InputAction.SingleTap)
        await asyncio.sleep(0.1)

    async def _siri_start(self) -> Mapping[str, Any]:
        response = await self._send_command("_siriStart", {})
        await asyncio.gather(*self.dispatch("_siriStart", response.get("_c", {})))
        return response

    async def _siri_stop(self) -> None:
        await self._send_command("_siriStop", {})
thiccaxe: You should be able to submit using the text commands.
albaintor: > You should be able to submit using the text commands.

I am not sure : maybe there is a command to submit input text (instead of moving cursor to submit button and click). However is there a dedicated command to send the submitted text as a siri command (when accessibility setting is disabled) ? Not so sure
albaintor: Hi @postlund will you have time to review the last updates ?
Thank you
postlund: Sorry for the delay, I will review it later tonight before going to bed.
postlund: Left a few comments, think that is it.
albaintor: Hi,

I have not added the swipe sequences "templates" as there seem to be several ways to do it. 
Also if we want to cover oblique gestures (from bottom left to top right) it would multiply the templates but I like this idea of bringing some kind of helper methods. 
What could be done instead is to define all the possible possitions : LEFT, RIGHT, TOP, BOTTOM, TOP_LEFT, TOP_RIGHT, BOTTOM_LEFT, BOTTOM_RIGHT and then having a method which takes 2 parameters : start & end coordinates?
thiccaxe: If singular cardinal directions are wanted through this api you must only move 200 pixels or so in that cardinal direction. You can't guarantee that a longer swipe will move, say, 2-3 ui elements because the Apple TV user can change the input sensitivity in settings. The better options is many smaller swipes-if swipes are absolutely needed. Might as well use the hidC option instead
albaintor: > If singular cardinal directions are wanted through this api you must only move 200 pixels or so in that cardinal direction. You can't guarantee that a longer swipe will move, say, 2-3 ui elements because the Apple TV user can change the input sensitivity in settings. The better options is many smaller swipes-if swipes are absolutely needed. Might as well use the hidC option instead

The touch range is 1000px x 1000px : so this is true if you are using the apple remote, however from an iphone/ipad you have a bigger touch window so more precise gestures. The final goal is to make touch gestures available from third party devices like home assistant or a remote with a touchscreen
thiccaxe: > The touch range is 1000px x 1000px : so this is true if you are using the apple remote, however from an iphone/ipad you have a bigger touch window so more precise gestures. 

I agree. However there can be some unexpected behavior as well (writing here for future users of this api). In tvOs a single swipe =/= many small swipes. For example, some ui elements are bounded in a no swipe zone, where you cannot exit the zone as a second motion in a single, continuous swipe, but can only do so as the first motion in a continuous swipe.
thiccaxe: > The final goal is to make touch gestures available from third party devices like home assistant or a remote with a touchscreen

For the remote situation, I wonder if translating the user's interactions directly to touch packets might be a better solution than another layer of abstraction/api through pyatv.
albaintor: > > The final goal is to make touch gestures available from third party devices like home assistant or a remote with a touchscreen
> 
> For the remote situation, I wonder if translating the user's interactions directly to touch packets might be a better solution than another layer of abstraction/api through pyatv.

You're right, I think we will have both needs : I am currently working for this project https://www.unfoldedcircle.com/ and both functions will be useful: 

1. Touch swipes horizontal (mostly) and vertical : it will address issues with inconsistent handle of fast forward/rewind among apps on apple TV.
2. Touchpad on the touch screen or with the "airmouse" feature of the remote -> for this one I will use the hid (lowlevel) events directly

postlund: I believe we have a lot of valid points here, but maybe we should focus on the essentials in this PR and open new PRs with improvements. I will try to review the last few changes and have this merged.
postlund: For later reference... size of the track pad can be saved as settings in storage.
thiccaxe: On that note, I will test if the trackpad size actually makes a difference, maybe through atvproxy.
albaintor: All done here !
postlund: @albaintor There are a few linting stuff to fix.
albaintor: > @albaintor There are a few linting stuff to fix.


albaintor: Hi,
it was sorting issues. Should be fixed now thanks
postlund: @albaintor I don't think you managed to push any changes?
albaintor: > @albaintor I don't think you managed to push any changes?

Are you sure ? I have pushed everything on my side
For example : pyatv/interfaces.py:1284 --> method renamed click instead of touch_click (N-1 commit)
And linting fix pushed from https://github.com/albaintor/pyatv/commit/d307a8cdd39aae5388b05c9915c55e9873831dfc
For example : pyatv/core/facade.py
![image](https://github.com/user-attachments/assets/1d6f89eb-e844-4634-9087-3eaffcd5f747)


postlund: @albaintor I just noted that there are no pushed changes after my last comment according to the PR. There are also still checks that are failing, you can check that below.
albaintor: Hi I think I know what is going on : something is wrong on the build action
![image](https://github.com/user-attachments/assets/365e6dcd-4bd1-479c-9971-b84802e673e1)

The black command fails with the new "match case" code in tests/fake_device/companion.py
The target python should be added in the black : -t py310 or whatever
postlund: > Hi I think I know what is going on : something is wrong on the build action ![image](https://private-user-images.githubusercontent.com/118518828/350907625-365e6dcd-4bd1-479c-9971-b84802e673e1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIzMjU3NzAsIm5iZiI6MTcyMjMyNTQ3MCwicGF0aCI6Ii8xMTg1MTg4MjgvMzUwOTA3NjI1LTM2NWU2ZGNkLTRiZDEtNDc5Yy05OTcxLWI4NDgwMmU2NzNlMS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDczMFQwNzQ0MzBaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zNWM2ODMxNTExYTg0ODg2OTNhZjRiM2I1NzUyOWJkOGU3Y2YyOGM0NjMzYjQ1NDcxNGIzNGY4NDYyNzAwMDU3JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.GlGe_j7gVikE4Z-MJzXSVryV_e2j5GwFl6DuyNREhD0)
> 
> The black command fails with the new "match case" code in tests/fake_device/companion.py The target python should be added in the black : -t py310 or whatever
> 
> EDIT : Addin `black --target-version py310 --fast --check .` within chickn.yaml would resolve the problem in order to accept python >3.10 match / case routines. But it also raises new linting issues. Just want to be sure what do you want to do from there See ref [psf/black#2242](https://github.com/psf/black/issues/2242)

Ah, I don't want to drop support for older Python versions just because of this. Just revert to an if-statement instead. You can use `./scripts/chickn.py -t fixup` to deal with linting properly (it's what's used by CI).
albaintor: > > Hi I think I know what is going on : something is wrong on the build action ![image](https://private-user-images.githubusercontent.com/118518828/350907625-365e6dcd-4bd1-479c-9971-b84802e673e1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIzMjU3NzAsIm5iZiI6MTcyMjMyNTQ3MCwicGF0aCI6Ii8xMTg1MTg4MjgvMzUwOTA3NjI1LTM2NWU2ZGNkLTRiZDEtNDc5Yy05OTcxLWI4NDgwMmU2NzNlMS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDczMFQwNzQ0MzBaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zNWM2ODMxNTExYTg0ODg2OTNhZjRiM2I1NzUyOWJkOGU3Y2YyOGM0NjMzYjQ1NDcxNGIzNGY4NDYyNzAwMDU3JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.GlGe_j7gVikE4Z-MJzXSVryV_e2j5GwFl6DuyNREhD0)
> > The black command fails with the new "match case" code in tests/fake_device/companion.py The target python should be added in the black : -t py310 or whatever
> > EDIT : Addin `black --target-version py310 --fast --check .` within chickn.yaml would resolve the problem in order to accept python >3.10 match / case routines. But it also raises new linting issues. Just want to be sure what do you want to do from there See ref [psf/black#2242](https://github.com/psf/black/issues/2242)
> 
> Ah, I don't want to drop support for older Python versions just because of this. Just revert to an if-statement instead. You can use `./scripts/chickn.py -t fixup` to deal with linting properly (it's what's used by CI).

Hi, this is all done including linting
There are some linting warnings but not related to my modifications
albaintor: > As long as the linting fails, you must fix it. You can suppress too-many-branches and you probably have to move the existing suppression for logging.TRAFFIC.

It should be okay this time
albaintor: All tests passed !
postlund: Yes! 🥳 

Patch:
diff --git a/docs/documentation/protocols.md b/docs/documentation/protocols.md
index dad56070..fda8605d 100644
--- a/docs/documentation/protocols.md
+++ b/docs/documentation/protocols.md
@@ -1046,6 +1046,65 @@ Example: Put device to sleep:
 {'_c': {}, '_t': 3, '_x': 123}
 ```
 
+#### Touch gestures
+
+Additional information about slide gestures : slide gestures are handled with a succession of events with about 20ms between each requests
+ 
+1. First event with _tPh=1 (press mode)
+2. N requests with _tPh = 3 (where N*100 ms = duration of the gesture), with _cx and _cy coordinates changing at each request
+3. One last request with _tPh=4 when released
+
+To be noted :
+- _cx and _cy coordinates must be in the range [0,1000] which is set in a startup event _touchStart :
+```javascript
+// Received during initialization
+ {'_i': '_touchStart', '_x': 1865081428, '_btHP': False, '_c': {'_height': 1000.0, '_tFl': 0, '_width': 1000.0}, '_t': 2}
+ ```
+- _ns = timestamp in nanoseconds (probably based on device boot time)
+- when reaching the edge of the touch area, a release event should be sent with a new press event otherwise the cursor will move in the opposite way 
+
+
+#### Single tap
+
+3 requests have to be sent to simulate tap gesture on the touch pad : 2 commands requests (_hidC) and 1 event request (_hidT)
+- 2 requests with _i = _hidC and in the additional arguments structure _C :
+  - the button pressed : _hidC = 6 for touch pad click
+  - the action mode : _hBtS = 1 for press, and the second request with _hBtS = 2 for release
+-  1 event with additional arguments as followed :
+  - _cx and _cy (x,y coordinates) set to max : 1000
+  - _tPh : action mode of the touchpad set to 5
+
+```javascript
+// Request
+{'_i': '_hidC', '_x': 1984212224, '_btHP': False, '_c': {'_hBtS': 1, '_hidC': 6}, '_t': 2}
+{'_i': '_hidC', '_x': 1984212225, '_btHP': False, '_c': {'_hBtS': 2, '_hidC': 6}, '_t': 2}
+{'_i': '_hidT', '_x': 1984212226, '_c': {'_ns': 713243707438041, '_tFg': 1, '_cx': 1000, '_tPh': 5, '_cy': 1000}, '_t': 1}
+
+// Response
+R: {'_c': {}, '_t': 3, '_x': 1984212224}
+R: {'_c': {}, '_t': 3, '_x': 1984212225}
+```
+
+
+#### Gestures
+
+Touch gestures are a series of events (_i = "_hidT", _t = 1) sent every few milliseconds (~20ms) with updated x,y coordinates
+- 1 start event with _tPh = 1 (pressed event)
+- N events with _tPh = 3 (hold)
+- 1 end event with _tPh = 4 (released)
+
+```javascript
+// Request
+{'_i': '_hidT', '_x': 588648840, '_c': {'_ns': 713018028759791, '_tFg': 1, '_cx': 500, '_tPh': 1, '_cy': 800}, '_t': 1}
+{'_i': '_hidT', '_x': 588648841, '_c': {'_ns': 713018124653791, '_tFg': 1, '_cx': 506, '_tPh': 3, '_cy': 789}, '_t': 1}
+{'_i': '_hidT', '_x': 588648842, '_c': {'_ns': 713018141323791, '_tFg': 1, '_cx': 520, '_tPh': 3, '_cy': 767}, '_t': 1}
+{'_i': '_hidT', '_x': 588648843, '_c': {'_ns': 713018157994791, '_tFg': 1, '_cx': 520, '_tPh': 3, '_cy': 758}, '_t': 1}
+...
+{'_i': '_hidT', '_x': 588648930, '_c': {'_ns': 713019612448791, '_tFg': 1, '_cx': 587, '_tPh': 3, '_cy': 128}, '_t': 1}
+{'_i': '_hidT', '_x': 588648931, '_c': {'_ns': 713019616615791, '_tFg': 1, '_cx': 593, '_tPh': 4, '_cy': 134}, '_t': 1}
+```
+
+
 #### System Status
 
 A system can be in one of the following states:
diff --git a/docs/documentation/supported_features.md b/docs/documentation/supported_features.md
index 252cc984..f30cc1d1 100644
--- a/docs/documentation/supported_features.md
+++ b/docs/documentation/supported_features.md
@@ -139,6 +139,7 @@ app and power related functions.
 * Turn on/off device ({% include api i="interface.Power.turn_on" %},
   {% include api i="interface.Power.turn_off" %})
 * Remote control (see {% include api i="interface.RemoteControl" %} for supported buttons)
+* Touch gestures and click
 
 ### Limitations and notes
 
diff --git a/pyatv/const.py b/pyatv/const.py
index d0d84091..6e2b5d5c 100644
--- a/pyatv/const.py
+++ b/pyatv/const.py
@@ -434,3 +434,21 @@ class FeatureName(Enum):
 
     TextSet = 54
     """Replace text in virtual keyboard."""
+
+    Swipe = 63
+    """Touch swipe from given coordinates and duration."""
+
+    Action = 64
+    """Touch event to given coordinates."""
+
+    Click = 65
+    """Touch click command."""
+
+
+class TouchAction(Enum):
+    """Touch action constants."""
+
+    Press = 1
+    Hold = 3
+    Release = 4
+    Click = 5
diff --git a/pyatv/core/facade.py b/pyatv/core/facade.py
index e223d02e..c1740943 100644
--- a/pyatv/core/facade.py
+++ b/pyatv/core/facade.py
@@ -18,7 +18,7 @@ from queue import Queue
 from typing import Any, Callable, Dict, List, Optional, Set, Tuple, Union, cast
 
 from pyatv import const, exceptions, interface
-from pyatv.const import FeatureName, FeatureState, InputAction, Protocol
+from pyatv.const import FeatureName, FeatureState, InputAction, Protocol, TouchAction
 from pyatv.core import CoreStateDispatcher, SetupData, StateMessage, UpdatedState
 from pyatv.core.relayer import Relayer
 from pyatv.interface import OutputDevice
@@ -601,6 +601,41 @@ class FacadePushUpdater(
             self.listener.playstatus_error(updater, exception)
 
 
+class FacadeTouchGestures(Relayer, interface.TouchGestures):
+    """Facade implementation for touch gestures handling."""
+
+    def __init__(self, core_dispatcher: CoreStateDispatcher):
+        """Initialize a new FacadeTouchGestures instance."""
+        Relayer.__init__(self, interface.TouchGestures, DEFAULT_PRIORITIES)
+        interface.TouchGestures.__init__(self)
+
+    @shield.guard
+    async def swipe(
+        self, start_x: int, start_y: int, end_x: int, end_y: int, duration_ms: int
+    ) -> None:
+        """Generate a touch gesture from start to end x,y coordinates."""
+        return await self.relay("swipe")(
+            start_x=start_x,
+            start_y=start_y,
+            end_x=end_x,
+            end_y=end_y,
+            duration_ms=duration_ms,
+        )
+
+    @shield.guard
+    async def action(self, x: int, y: int, mode: TouchAction) -> None:
+        """Generate a touch event to end x,y coordinates."""
+        return await self.relay("action")(x=x, y=y, mode=mode)
+
+    @shield.guard
+    async def click(self, action: InputAction):
+        """Send a touch click.
+
+        :param action: action mode, single tap (0), double tap (1), or hold (2).
+        """
+        return await self.relay("click")(action=action)
+
+
 class FacadeAppleTV(interface.AppleTV):
     """Facade implementation of the external interface."""
 
@@ -632,6 +667,7 @@ class FacadeAppleTV(interface.AppleTV):
             interface.UserAccounts: FacadeUserAccounts(),
             interface.Audio: FacadeAudio(core_dispatcher),
             interface.Keyboard: FacadeKeyboard(core_dispatcher),
+            interface.TouchGestures: FacadeTouchGestures(core_dispatcher),
         }
         self._settings = settings
         self._shield_everything()
@@ -833,6 +869,12 @@ class FacadeAppleTV(interface.AppleTV):
         """Return keyboard interface."""
         return cast(interface.Keyboard, self._interfaces[interface.Keyboard])
 
+    @property  # type: ignore
+    @shield.guard
+    def touch(self) -> interface.TouchGestures:
+        """Return touch gestures interface."""
+        return cast(interface.TouchGestures, self._interfaces[interface.TouchGestures])
+
     def state_was_updated(self) -> None:
         """Call when state was updated.
 
diff --git a/pyatv/interface.py b/pyatv/interface.py
index 4affd4bb..8a8d56f0 100644
--- a/pyatv/interface.py
+++ b/pyatv/interface.py
@@ -37,6 +37,7 @@ from pyatv.const import (
     OperatingSystem,
     PairingRequirement,
     Protocol,
+    TouchAction,
 )
 from pyatv.settings import Settings
 from pyatv.support import prettydataclass
@@ -1249,6 +1250,45 @@ class Keyboard(ABC, StateProducer):
         raise exceptions.NotSupportedError()
 
 
+class TouchGestures(ABC):
+    """Base class for touch gestures."""
+
+    @feature(63, "Swipe", "Swipe gesture from given coordinates and duration.")
+    async def swipe(
+        self, start_x: int, start_y: int, end_x: int, end_y: int, duration_ms: int
+    ) -> None:
+        """Generate a swipe gesture.
+
+         From start to end x,y coordinates (in range [0,1000])
+         in a given time (in milliseconds).
+
+        :param start_x: Start x coordinate
+        :param start_y: Start y coordinate
+        :param end_x: End x coordinate
+        :param end_y: Endi x coordinate
+        :param duration_ms: Time in milliseconds to reach the end coordinates
+        """
+        raise exceptions.NotSupportedError()
+
+    @feature(64, "TouchAction", "Touch event to given coordinates.")
+    async def action(self, x: int, y: int, mode: TouchAction) -> None:
+        """Generate a touch event to x,y coordinates (in range [0,1000]).
+
+        :param x: x coordinate
+        :param y: y coordinate
+        :param mode: touch mode (1: press, 3: hold, 4: release)
+        """
+        raise exceptions.NotSupportedError()
+
+    @feature(65, "TouchClick", "Touch click command.")
+    async def click(self, action: InputAction):
+        """Send a touch click.
+
+        :param action: action mode single tap (0), double tap (1), or hold (2)
+        """
+        raise exceptions.NotSupportedError()
+
+
 class BaseConfig(ABC):
     """Representation of a device configuration.
 
@@ -1524,3 +1564,8 @@ class AppleTV(ABC, StateProducer[DeviceListener]):
     @abstractmethod
     def keyboard(self) -> Keyboard:
         """Return keyboard interface."""
+
+    @property
+    @abstractmethod
+    def touch(self) -> TouchGestures:
+        """Return touch gestures interface."""
diff --git a/pyatv/protocols/companion/__init__.py b/pyatv/protocols/companion/__init__.py
index ad7e5f58..dcd621ab 100644
--- a/pyatv/protocols/companion/__init__.py
+++ b/pyatv/protocols/companion/__init__.py
@@ -15,6 +15,7 @@ from pyatv.const import (
     PairingRequirement,
     PowerState,
     Protocol,
+    TouchAction,
 )
 from pyatv.core import Core, MutableService, SetupData, UpdatedState, mdns
 from pyatv.core.scan import (
@@ -35,6 +36,7 @@ from pyatv.interface import (
     PairingHandler,
     Power,
     RemoteControl,
+    TouchGestures,
     UserAccount,
     UserAccounts,
 )
@@ -140,6 +142,9 @@ SUPPORTED_FEATURES = set(
         FeatureName.TextClear,
         FeatureName.TextAppend,
         FeatureName.TextSet,
+        FeatureName.Swipe,
+        FeatureName.Action,
+        FeatureName.Click,
     ]
     # Remote control (playback, i.e. Media Control)
     + list(MEDIA_CONTROL_MAP.keys())
@@ -461,6 +466,47 @@ class CompanionKeyboard(Keyboard):
         await self.api.text_input_command(text, clear_previous_input=True)
 
 
+class CompanionTouchGestures(TouchGestures):
+    """Implementation of touch gesture API."""
+
+    def __init__(self, api: CompanionAPI) -> None:
+        """Initialize a new CompanionTouchGeatures."""
+        self.api = api
+
+    # pylint: disable=invalid-name
+    async def swipe(
+        self, start_x: int, start_y: int, end_x: int, end_y: int, duration_ms: int
+    ) -> None:
+        """Generate a touch swipe.
+
+         From start to end x,y coordinates (in range [0,1000])
+         in a given time (in milliseconds).
+
+        :param start_x: Start x coordinate
+        :param start_y: Start y coordinate
+        :param end_x: End x coordinate
+        :param end_y: Endi x coordinate
+        :param duration_ms: Time in milliseconds to reach the end coordinates
+        """
+        await self.api.swipe(start_x, start_y, end_x, end_y, duration_ms)
+
+    async def action(self, x: int, y: int, mode: TouchAction):
+        """Generate a touch event to x,y coordinates (in range [0,1000]).
+
+        :param x: x coordinate
+        :param y: y coordinate
+        :param mode: touch mode (1: press, 3: hold, 4: release)
+        """
+        await self.api.action(x, y, mode)
+
+    async def click(self, action: InputAction):
+        """Send a touch click.
+
+        :param action: action mode single tap (0), double tap (1), or hold (2)
+        """
+        await self.api.click(action)
+
+
 class CompanionFeatures(Features):
     """Implementation of supported feature functionality."""
 
@@ -565,6 +611,7 @@ def setup(core: Core) -> Generator[SetupData, None, None]:
         RemoteControl: CompanionRemoteControl(api),
         Audio: CompanionAudio(api, core),
         Keyboard: CompanionKeyboard(api, core),
+        TouchGestures: CompanionTouchGestures(api),
     }
 
     async def _connect() -> bool:
diff --git a/pyatv/protocols/companion/api.py b/pyatv/protocols/companion/api.py
index 279b185a..0adf6e27 100644
--- a/pyatv/protocols/companion/api.py
+++ b/pyatv/protocols/companion/api.py
@@ -4,11 +4,13 @@ import asyncio
 from enum import Enum
 import logging
 from random import randint
+import time
 from typing import Any, Dict, List, Mapping, Optional, cast
 
 from pyatv import exceptions
 from pyatv.auth.hap_pairing import parse_credentials
 from pyatv.auth.hap_srp import SRPAuthHandler
+from pyatv.const import InputAction, TouchAction
 from pyatv.core import Core
 from pyatv.core.protocol import MessageDispatcher
 from pyatv.protocols.companion import keyed_archiver
@@ -83,6 +85,9 @@ class SystemStatus(Enum):
     Idle = 0x04  # NB: Not verified
 
 
+TOUCHPAD_WIDTH = 1000.0  # Touchpad width
+TOUCHPAD_HEIGHT = 1000.0  # Touchpad height
+TOUCHPAD_DELAY_MS = 16  # Delay between touch events in ms
 # pylint: enable=invalid-name
 
 
@@ -99,6 +104,7 @@ class CompanionAPI(
         self._protocol: Optional[CompanionProtocol] = None
         self._subscribed_events: List[str] = []
         self.sid: int = 0
+        self._base_timestamp = time.time_ns()
 
     async def disconnect(self):
         """Disconnect from companion device."""
@@ -113,6 +119,7 @@ class CompanionAPI(
             # going to bother with that and just swallow the error.
             await self._session_stop()
 
+            await self._touch_stop()
             await self._text_input_stop()
         except Exception as ex:
             _LOGGER.debug("Ignoring error during disconnect: %s", ex)
@@ -144,6 +151,7 @@ class CompanionAPI(
         await self._protocol.start()
 
         await self.system_info()
+        await self._touch_start()
         await self._session_start()
         await self._text_input_start()
 
@@ -283,6 +291,90 @@ class CompanionAPI(
             "_hidC", {"_hBtS": 1 if down else 2, "_hidC": command.value}
         )
 
+    async def hid_event(self, x: int, y: int, mode: TouchAction) -> None:
+        """Send a HID command."""
+        x = max(x, 0)
+        y = max(y, 0)
+        x = min(x, int(TOUCHPAD_WIDTH))
+        y = min(y, int(TOUCHPAD_HEIGHT))
+        await self._send_event(
+            identifier="_hidT",
+            content={
+                "_ns": (time.time_ns() - self._base_timestamp),
+                "_tFg": 1,
+                "_cx": x,
+                "_tPh": mode.value,
+                "_cy": y,
+            },
+        )
+
+    async def swipe(
+        self, start_x: int, start_y: int, end_x: int, end_y: int, duration_ms: int
+    ):
+        """Generate a swipe gesture.
+
+         From start to end x,y coordinates (in range [0,1000])
+         in a given time (in milliseconds).
+
+        :param start_x: Start x coordinate
+        :param start_y: Start y coordinate
+        :param end_x: End x coordinate
+        :param end_y: Endi x coordinate
+        :param duration_ms: Time in milliseconds to reach the end coordinates
+        """
+        end_time = time.time_ns() + duration_ms * 1000000
+        x: float = float(start_x)
+        y: float = float(start_y)
+        await self.hid_event(int(x), int(y), TouchAction.Press)
+        sleep_time = float(TOUCHPAD_DELAY_MS / 1000)
+        current_time = time.time_ns()
+        while current_time < end_time:
+            x = x + float(end_x - x) * TOUCHPAD_DELAY_MS * 1000000 / (
+                end_time - current_time
+            )
+            y = y + float(end_y - y) * TOUCHPAD_DELAY_MS * 1000000 / (
+                end_time - current_time
+            )
+            x = max(x, 0)
+            y = max(y, 0)
+            x = min(x, TOUCHPAD_WIDTH)
+            y = min(y, TOUCHPAD_HEIGHT)
+            await self.hid_event(int(x), int(y), TouchAction.Hold)
+            await asyncio.sleep(sleep_time)
+            current_time = time.time_ns()
+        await self.hid_event(end_x, end_y, TouchAction.Release)
+
+    async def action(self, x: int, y: int, mode: TouchAction):
+        """Generate a touch event to x,y coordinates (in range [0,1000]).
+
+        :param x: x coordinate
+        :param y: y coordinate
+        :param mode: touch mode (1: press, 3: hold, 4: release)
+        """
+        await self.hid_event(x, y, mode)
+
+    async def click(self, action: InputAction):
+        """Send a touch click.
+
+        :param action: action mode single tap (0), double tap (1), or hold (2)
+        """
+        if action in [InputAction.SingleTap, InputAction.DoubleTap]:
+            count = 1 if action == InputAction.SingleTap else 2
+            for _i in range(count):
+                await self._send_command("_hidC", {"_hBtS": 1, "_hidC": 6})
+                await asyncio.sleep(0.02)
+                await self._send_command("_hidC", {"_hBtS": 2, "_hidC": 6})
+                await self.hid_event(
+                    int(TOUCHPAD_WIDTH), int(TOUCHPAD_HEIGHT), TouchAction.Click
+                )
+        else:  # Hold
+            await self._send_command("_hidC", {"_hBtS": 1, "_hidC": 6})
+            await asyncio.sleep(1)
+            await self._send_command("_hidC", {"_hBtS": 2, "_hidC": 6})
+            await self.hid_event(
+                int(TOUCHPAD_WIDTH), int(TOUCHPAD_HEIGHT), TouchAction.Click
+            )
+
     async def mediacontrol_command(
         self, command: MediaControlCommand, args: Optional[Mapping[str, Any]] = None
     ) -> Mapping[str, Any]:
@@ -351,3 +443,16 @@ class CompanionAPI(
             raise exceptions.ProtocolError("missing content")
 
         return SystemStatus(content["state"])
+
+    async def _touch_start(self) -> Mapping[str, Any]:
+        """Subscribe to touch gestures."""
+        self._base_timestamp = time.time_ns()
+        response = await self._send_command(
+            "_touchStart",
+            {"_height": TOUCHPAD_HEIGHT, "_tFl": 0, "_width": TOUCHPAD_WIDTH},
+        )
+        return response
+
+    async def _touch_stop(self) -> None:
+        """Unsubscribe touch gestures."""
+        await self._send_command("_touchStop", {"_i": 1})
diff --git a/pyatv/scripts/atvremote.py b/pyatv/scripts/atvremote.py
index 1b9843a7..977c38c4 100644
--- a/pyatv/scripts/atvremote.py
+++ b/pyatv/scripts/atvremote.py
@@ -21,6 +21,7 @@ from pyatv.const import (
     Protocol,
     RepeatState,
     ShuffleState,
+    TouchAction,
 )
 from pyatv.interface import BaseConfig, BaseService, Storage, retrieve_commands
 from pyatv.scripts import (
@@ -104,6 +105,7 @@ class GlobalCommands:
         _print_commands("Apps", interface.Apps)
         _print_commands("User Accounts", interface.UserAccounts)
         _print_commands("Global", self.__class__)
+        _print_commands("Touch", interface.TouchGestures)
 
         return 0
 
@@ -123,6 +125,7 @@ class GlobalCommands:
             interface.Apps,
             interface.Audio,
             interface.Keyboard,
+            interface.TouchGestures,
             self.__class__,
             DeviceCommands,
         ]
@@ -829,10 +832,21 @@ def _extract_command_with_args(cmd):
             return [ShuffleState(args[0])]
         if cmd == "set_repeat":
             return [RepeatState(args[0])]
-        if cmd in ["up", "down", "left", "right", "select", "menu", "home"]:
+        if cmd in [
+            "up",
+            "down",
+            "left",
+            "right",
+            "select",
+            "menu",
+            "home",
+            "touch_click",
+        ]:
             return [InputAction(args[0])]
         if cmd == "set_volume":
             return [float(args[0])]
+        if cmd == "touch_action":
+            return [args[0], args[1], TouchAction(args[2])]
         return args
 
     equal_sign = cmd.find("=")
@@ -870,6 +884,7 @@ async def _handle_commands(args, config, storage: Storage, loop):
 
 
 # pylint: disable=too-many-return-statements disable=too-many-locals
+# pylint: disable=too-many-branches
 async def _handle_device_command(args, cmd, atv, storage: Storage, loop):
     device = retrieve_commands(DeviceCommands)
     settings = retrieve_commands(SettingsCommands)
@@ -883,6 +898,7 @@ async def _handle_device_command(args, cmd, atv, storage: Storage, loop):
     user_accounts = retrieve_commands(interface.UserAccounts)
     audio = retrieve_commands(interface.Audio)
     keyboard = retrieve_commands(interface.Keyboard)
+    touch = retrieve_commands(interface.TouchGestures)
 
     # Parse input command and argument from user
     cmd, cmd_args = _extract_command_with_args(cmd)
@@ -927,6 +943,9 @@ async def _handle_device_command(args, cmd, atv, storage: Storage, loop):
     if cmd in user_accounts:
         return await _exec_command(atv.user_accounts, cmd, True, *cmd_args)
 
+    if cmd in touch:
+        return await _exec_command(atv.touch, cmd, True, *cmd_args)
+
     _LOGGER.error("Unknown command: %s", cmd)
     return 1
 


Test patch:
diff --git a/tests/fake_device/companion.py b/tests/fake_device/companion.py
index 1fa97cba..f910e9f6 100644
--- a/tests/fake_device/companion.py
+++ b/tests/fake_device/companion.py
@@ -1,12 +1,13 @@
 """Fake Companion Apple TV for tests."""
 
 import asyncio
+from dataclasses import dataclass
 from enum import IntFlag, auto
 import logging
 import plistlib
 from typing import Any, Dict, List, Mapping, Optional, Set
 
-from pyatv.const import KeyboardFocusState
+from pyatv.const import KeyboardFocusState, TouchAction
 from pyatv.protocols.companion import (
     HidCommand,
     MediaControlCommand,
@@ -57,6 +58,14 @@ MEDIA_CONTROL_MAP = {
 }
 
 
+@dataclass
+class HidEvent:
+    press_mode: TouchAction
+    x: int
+    y: int
+    ns: int
+
+
 class CompanionServiceFlags(IntFlag):
     """Flags used to alter fake service behavior."""
 
@@ -92,6 +101,9 @@ class FakeCompanionState:
         self._rti_focus_state: KeyboardFocusState = KeyboardFocusState.Focused
         self.rti_text: Optional[str] = INITIAL_RTI_TEXT
         self.rti_session_uuid: Optional[bytes] = None
+        self.touch_event: HidEvent | None = None
+        self.touch_width = 0
+        self.touch_height = 0
 
     def is_supported(self, flag: CompanionServiceFlags) -> bool:
         """Return if a feature is supported."""
@@ -172,6 +184,10 @@ class FakeCompanionState:
         else:
             return {}
 
+    @property
+    def touch_event_state(self) -> HidEvent:
+        return self.touch_event
+
 
 class FakeCompanionServiceFactory:
     def __init__(self, state, app, loop):
@@ -388,6 +404,49 @@ class FakeCompanionService(CompanionServerAuth, asyncio.Protocol):
 
         self.send_response(message, {})
 
+    def handle__touchstart(self, message):
+        width = message["_c"]["_width"]
+        height = message["_c"]["_height"]
+        _LOGGER.debug(
+            "Touch start command received with touchpad width %s and height %s",
+            width,
+            height,
+        )
+        self.state.touch_width = width
+        self.state.touch_height = height
+        if (
+            not width
+            or width < 0
+            or width > 1000
+            or not height
+            or height < 0
+            or height > 1000
+        ):
+            self.send_error(message, "Invalid touchpad width or height")
+        else:
+            self.send_response(message, {})
+
+    def handle__touchstop(self, message):
+        _LOGGER.debug("Touch stop command received")
+        self.send_response(message, {})
+
+    def handle__hidt(self, message):
+        press_mode: int = message["_c"]["_tPh"]
+        ns = message["_c"]["_ns"]
+        cx = message["_c"]["_cx"]
+        cy = message["_c"]["_cy"]
+        if press_mode == TouchAction.Press:
+            _LOGGER.debug("Touch event press to (%s, %s) at time %s", cx, cy, ns)
+        elif TouchAction.Hold:
+            _LOGGER.debug("Touch event move to (%s, %s) at time %s", cx, cy, ns)
+        elif press_mode == TouchAction.Release:
+            _LOGGER.debug("Touch event release to (%s, %s) at time %s", cx, cy, ns)
+        elif press_mode == TouchAction.Click:
+            _LOGGER.debug("Touch event click to (%s, %s) at time %s", cx, cy, ns)
+        else:
+            _LOGGER.warning("Touch event mode not supported %s", press_mode)
+        self.state.action = HidEvent(TouchAction(press_mode), cx, cy, ns)
+
     def handle__mcc(self, message):
         args = {}
         mcc = MediaControlCommand(message["_c"]["_mcc"])
diff --git a/tests/protocols/companion/test_companion_functional.py b/tests/protocols/companion/test_companion_functional.py
index afd87a93..796c0c5e 100644
--- a/tests/protocols/companion/test_companion_functional.py
+++ b/tests/protocols/companion/test_companion_functional.py
@@ -9,7 +9,13 @@ import pytest
 import pyatv
 from pyatv import exceptions
 from pyatv.conf import AppleTV, ManualService
-from pyatv.const import KeyboardFocusState, PowerState, Protocol
+from pyatv.const import (
+    InputAction,
+    KeyboardFocusState,
+    PowerState,
+    Protocol,
+    TouchAction,
+)
 from pyatv.interface import App, FeatureName, FeatureState, UserAccount
 from pyatv.protocols.companion.api import SystemStatus
 
@@ -335,3 +341,23 @@ async def test_power_state_availability(
     assert atv.power.power_state == expecter_power_state
 
     await asyncio.gather(*atv.close())
+
+
+async def test_touch(companion_client, companion_state):
+    await companion_client.touch.swipe(0, 0, 800, 800, 200)
+    await until(
+        lambda: companion_state.action
+        and companion_state.action.x == 800
+        and companion_state.action.y == 800
+        and companion_state.action.press_mode == TouchAction.Release
+    )
+
+
+async def test_touch_click(companion_client, companion_state):
+    await companion_client.touch.click(InputAction.SingleTap)
+    assert companion_state.latest_button == "select"
+
+
+async def test_touch_hold(companion_client, companion_state):
+    await companion_client.touch.click(InputAction.Hold)
+    assert companion_state.latest_button == "select"