Repository: espanso/espanso (language: rust)

Issue:
MacOS - Clipboard seem to be limited to 2048 characters
Hello

**Describe the bug**
The clipboard seems to be limited to 2048 characters on MacOS Ventura

**To Reproduce**
Steps to reproduce the behavior:

I use this code for example : 
```
  # Inserts a markdown code block
  - triggers: [":code", "/code"]
    replace: "```\n{{clipboard}}\n```"
    vars:
    - name: "clipboard"
      type: "clipboard"
```

1. Copy long text
2. Paste with espanso replace

**Expected behavior**
Paste all text


**Logs**
```
10:44:56 [daemon(24085)] [INFO] configuration change detected, restarting worker process...
10:44:56 [worker(24332)] [INFO] engine eventloop has terminated, propagating exit event...
10:44:56 [worker(24332)] [INFO] waiting for engine exit mode...
10:44:56 [worker(24332)] [INFO] exiting worker process...
10:44:56 [daemon(24085)] [INFO] spawning the worker process...
10:44:56 [worker(24556)] [INFO] reading configs from: "/Users/papamica/Library/Application Support/espanso"
10:44:56 [worker(24556)] [INFO] reading packages from: "/Users/papamica/Library/Application Support/espanso/match/packages"
10:44:56 [worker(24556)] [INFO] using runtime dir: "/Users/papamica/Library/Caches/espanso"
10:44:56 [worker(24556)] [INFO] system info: Darwin v13.3.1 - kernel: 22.4.0
10:44:56 [worker(24556)] [INFO] using CocoaAppInfoProvider
10:44:56 [worker(24556)] [INFO] using CocoaSource
10:44:56 [worker(24556)] [INFO] using MacInjector
10:44:56 [worker(24556)] [INFO] binded to IPC unix socket: /Users/papamica/Library/Caches/espanso/espansoworkerv2.sock
10:44:56 [worker(24556)] [INFO] using CocoaClipboard
10:44:56 [worker(24556)] [INFO] monitoring the status of the daemon process
10:44:56 [worker(24556)] [INFO] monitoring the status of secure input
```
**Setup information**
- OS: MacOS (Intel) Ventura
- Version: espanso 2.1.8

Have a nice day



Discussion / hints:
I succeed to paste a >2048 text. (Debian 11, espanso 0.7.3, on Gnome and Xfce4) [Example code](https://gist.githubusercontent.com/QJoly/c6eed188408fcb2dc259a3b0fcb630e0/raw/b506c75979d8a0f43fb4a557a90dbeded52ba939/test-espanso.yaml) 

The problem is probably specific to macOS. 
Thanks @QJoly, I must have misspoken, I'm talking about the use of the clipboard, I've added a configuration example.
Sorry for the misunderstanding. However I get the same result.

I can paste a text of >2048 *(I used the same example as before)*

```yaml
      - triggers: [":code", "/code"]
        replace: "```\n{{clipboard}}\n```"
        vars:
        - name: "clipboard"
          type: "clipboard"
```

Input: [link](https://gist.github.com/QJoly/b5753924219ff39150d7c5bb36144d43)

Output:
```yaml
...
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, enim. Aliquam lorem ante, dapibus in, viverra quis, feugiat a, tellus. Phasellus viverra nulla ut metus varius laoreet. Quisque rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui. Etiam rhoncus. Maecenas tempus, tellus eget condimentum rhoncus, sem quam semper libero, sit amet adipiscing sem neque sed ipsum. Nam quam nunc, blandit vel, luctus pulvinar, hendrerit id, lorem. Maecenas nec odio et ante tincidunt tempus. Donec vitae sapien ut libero venenatis faucibus. Nullam quis ante. Etiam sit amet orci eget eros faucibus tincidunt. Duis leo. Sed fringilla mauris sit amet nibh. Donec sodales sagittis magna. Sed consequat, leo eget bibendum sodales, augue velit cursus nunc, quis gravida magna mi a libero. Fusce vulputate eleifend sapien. Vestibulum purus quam, scelerisque ut, mollis sed, nonummy id, metus. Nullam accumsan lorem in dui. Cras ultricies mi eu turpis hendrerit fringilla. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; In ac dui quis mi consectetuer lacinia. Nam pretium turpis et arcu. Duis arcu tortor, suscipit eget, imperdiet nec, imperdiet iaculis, ipsum. Sed aliquam ultrices mauris. Integer ante arcu, accumsan a, consectetuer eget, posuere ut, mauris. Praesent adipiscing. Phasellus ullamcorper ipsum rutrum nunc. Nunc nonummy metus. Vestibulum volutpat pretium libero. Cras id dui. Aenean ut eros et nisl sagittis vestibulum. Nullam nulla eros, ultricies sit amet, nonummy id, imperdiet.
...
```

Hello,

I just got exactly the same issue while using the clipboard feature on MacOS Sonoma 14.0 
Same issue on MacOS Sonoma 14.0 and Windows.

A workaround I am currently settling for:

Replace:

```
    - name: "clipboard"
      type: "clipboard"
```

With:

```
    - name: "clipboard"
      type: shell
       params:
          cmd: "pbpaste"
```

Be warned that this might introduce some latency, so there is a trade-off.
I've only tested this on Mac. A different command other than pbpaste should work for other OS.
Under Windows 10 the clipboard is also limited to 2048 characters.
Same problem here. On Windows 11.
> - name: "clipboard"
>       type: shell
>        params:
>           cmd: "pbpaste"

This would a solid workaround, but unfortunately it only works for the first paste. After that the clipboard is truncated after 2048 characters again. This remains a major problem.
> > * name: "clipboard"
> >   type: shell
> >   params:
> >   cmd: "pbpaste"
> 
> This would a solid workaround, but unfortunately it only works for the first paste. After that the clipboard is truncated after 2048 characters again. This remains a major problem.

@woerndl I encountered the same issue and use a similar workaround, but with `script` instead of `shell` (it avoids launching an extra shell and should be faster). On Windows, I use [gclip](https://github.com/golang-design/clipboard) in place of `pbpaste`.

When `backend: clipboard` is active, the second paste gets truncated to 2048 characters, likely due to the clipboard method. However, when `backend: inject` is used, it works repeatedly without truncation. The downside is that with large clipboards, it can freeze the computer temporarily since sending zillions of keystrokes is far from ideal.

Oooohh, interesting, I wasn't aware that there is performance to be gained while using `script`. I think you are right, it is faster! Way faster! There is a noticeable delay until the `shell` spins up, even on a M2 MacBook Air. While there is virtually none using `script`, thank you. And thank you not, now I have to refactor all my shell commands 🥲😄

Also `backend: inject` does indeed fix the problem in this combination. But, as you say, it is just to crazy. I'm gonna live with the situation for now. But this is of course a major problem and one that, in my opinion, deserves immediate attention in terms of triage as it touches on a functionality that is as core as it gets.

For others, pasting one time is possible (and performant) with:

```yaml
global_vars:
  # Attempt 2 to fix 2048 character problem
  - name: "clipboard"
    type: script
    params:
      args:
        - bash
        - -c
        - pbpaste
```
@woerndl You can use directly `pbpaste` like this :
```yaml
global_vars:
  # Attempt 2 to fix 2048 character problem
  - name: "clipboard"
    type: script
    params:
      args:
        - pbpaste
```
If espnso can't find `pbpaste` you can try with `full/path/to/pbpaste` in place.
Oh, thanks. For some reason this didn't work when I tried it, probably arranged the args wrong. Great!

I want to add that this issue is no mystery, it seems to be in fact just the defined buffer size, L42:

https://github.com/espanso/espanso/blob/9d19f1cf84f727d64b670923018e8d8d8351e40e/espanso-clipboard/src/cocoa/mod.rs#L40-L51

Affecting X11 (when not using `xclip`), Windows and macOS as all these implementations have the same limit.
There's no such limitation here (Linux Xfce). I have just used:
```yml
  - trigger: :test
    replace: '{{output}}'
    vars:
      - name: output
        type: clipboard
```
to paste 50000 bytes.        
@smeech I'm surprised because we can see the same hardcoded limit in [espanso-clipboard/src/x11/native/mod.rs](https://github.com/espanso/espanso/blob/9d19f1cf84f727d64b670923018e8d8d8351e40e/espanso-clipboard/src/x11/native/mod.rs#L43). 

But for linux there is also a `xclip` version in [espanso-clipboard/src/x11/xclip/mod.rs](https://github.com/espanso/espanso/blob/dev/espanso-clipboard/src/x11/xclip/mod.rs) that you probably use ?

For windows also the 2048 is hardcoded in [espanso-clipboard/src/win32/mod.rs](https://github.com/espanso/espanso/blob/9d19f1cf84f727d64b670923018e8d8d8351e40e/espanso-clipboard/src/win32/mod.rs#L40)
Ah, yes - I am using `x11_use_xclip_backend: true`.
I'm just ventilating here, as I've never developed anything with Rust and/or Objective-C (in the case of macOS). But it seems clear that the current clipboard implementation regularly creates headaches (at least for macOS, but likely for other implementations too). Issues range from destroying image data [#2059](https://github.com/espanso/espanso/issues/2059) ([#1226](https://github.com/espanso/espanso/issues/1226)), to failing when containing non-text [#2075](https://github.com/espanso/espanso/issues/2075).

Would it be a possibility to:

1. **Use `pbcopy`/`pbpaste` natively on macOS** similar to how `xclip` is used on Linux?
   - This could allow to [specify pasteboards](https://ss64.com/mac/pbcopy.html) (even [custom](https://developer.apple.com/documentation/appkit/nspasteboard/1528936-pasteboardwithuniquename?language=objc) ones?) and thus reduce cases where the content of the main pasteboard gets overwritten or corrupted. And also to avoid the hardcoded buffer limit altogether as the memory management is outsourced.

2. **Use a library such as `arboard` or `copypasta`** to abstract away platform-specific clipboard implementations?

3. **Keep the current handling**, but either:
   - Try to query the clipboard size beforehand and adjust the buffer accordingly, or
   - Dynamically adjust the buffer size with a larger starting limit to better handle various clipboard contents?

This could use a look from someone other than me who actually knows things about clipboard handling on OS level, haha.

Update: Removed cross reference to issues regarding the retainment of first letters as this has likely nothing todo with the clipboard handling.

Fix patch:
diff --git a/espanso-clipboard/src/win32/ffi.rs b/espanso-clipboard/src/win32/ffi.rs
index 601d8a80a..25aeedc40 100644
--- a/espanso-clipboard/src/win32/ffi.rs
+++ b/espanso-clipboard/src/win32/ffi.rs
@@ -21,6 +21,7 @@ use std::os::raw::c_char;
 
 #[link(name = "espansoclipboard", kind = "static")]
 extern "C" {
+  pub fn clipboard_get_length() -> i32;
   pub fn clipboard_get_text(buffer: *mut u16, buffer_size: i32) -> i32;
   pub fn clipboard_set_text(text: *const u16) -> i32;
   pub fn clipboard_set_image(image_path: *const u16) -> i32;
diff --git a/espanso-clipboard/src/win32/mod.rs b/espanso-clipboard/src/win32/mod.rs
index 61d8be2a0..78559468e 100644
--- a/espanso-clipboard/src/win32/mod.rs
+++ b/espanso-clipboard/src/win32/mod.rs
@@ -37,7 +37,13 @@ impl Win32Clipboard {
 
 impl Clipboard for Win32Clipboard {
   fn get_text(&self, _: &ClipboardOperationOptions) -> Option<String> {
-    let mut buffer: [u16; 2048] = [0; 2048];
+    // get the clipbard size
+    let length = unsafe { ffi::clipboard_get_length() };
+    if length <= 0 {
+      return None;
+    }
+    // allocate the buffer
+    let mut buffer: Vec<u16> = vec![0; length as usize];
     let native_result =
       unsafe { ffi::clipboard_get_text(buffer.as_mut_ptr(), (buffer.len() - 1) as i32) };
     if native_result > 0 {
diff --git a/espanso-clipboard/src/win32/native.cpp b/espanso-clipboard/src/win32/native.cpp
index af8d1d8db..f867579b9 100644
--- a/espanso-clipboard/src/win32/native.cpp
+++ b/espanso-clipboard/src/win32/native.cpp
@@ -42,6 +42,33 @@
 
 #include <Windows.h>
 
+int32_t clipboard_get_length() {
+    // Open the clipboard
+    if (!OpenClipboard(NULL)) {
+        return -1; // Failed to open clipboard
+    }
+
+    // Check if the clipboard contains text data
+    HANDLE hData = GetClipboardData(CF_UNICODETEXT);
+    if (hData == NULL) {
+        CloseClipboard();
+        return -1; // No text data in clipboard
+    }
+
+    // Get the size of the clipboard data
+    SIZE_T size = GlobalSize(hData);
+    if (size == 0) {
+        CloseClipboard();
+        return -1; // Failed to get size
+    }
+
+    // Close the clipboard
+    CloseClipboard();
+
+    // Return the size of the clipboard data
+    return static_cast<int32_t>(size);
+}
+
 int32_t clipboard_get_text(wchar_t *buffer, int32_t buffer_size)
 {
   int32_t result = 0;
diff --git a/espanso-clipboard/src/win32/native.h b/espanso-clipboard/src/win32/native.h
index ce49cea3a..2921b7f52 100644
--- a/espanso-clipboard/src/win32/native.h
+++ b/espanso-clipboard/src/win32/native.h
@@ -22,6 +22,7 @@
 
 #include <stdint.h>
 
+extern "C" int32_t clipboard_get_length();
 extern "C" int32_t clipboard_get_text(wchar_t * buffer, int32_t buffer_size);
 extern "C" int32_t clipboard_set_text(wchar_t * text);
 extern "C" int32_t clipboard_set_image(wchar_t * image);


Test patch: