Title: Named pipe between C++ and Python: unable to read from C++
Tags: python,c++,linux,macos,pipe

Question: My objective is to run a python and C++ process in lockstep using named pipes for communication. That is:

C++ process sends some message (delimited by \n) on PIPE_1
Python process reads message (also delimited by \n), sends response on PIPE_2
C++ reads message
Repeat 1..3

The messages contain arrays (possibly multi-dimensional), so I intend to use a library like protobuf to serialize the message to write to the pipe.
However, my C++ process blocks on std::getline and is unable to read from PIPE_2 until the python process is killed manually (at this point the C++ process gets the message). I suspected this might be a buffering issue, so I tried to call flush() as well as write \n to the pipe, but neither seems to work.
I am open to other solutions that are fast enough (I will be passing messages back and forth frequently), but would like to know where I'm going wrong here. My code is running on an Intel based Mac if that helps (the eventual deployment environment will be a linux one).
I rewrote a minimal version of the C++ code in python and this produces the correct behaviour (messages sent and received by both processes in lockstep, no blocking).
Below are minimally reproducible code samples:
Python process -
import errno
import os

pipe1 = '/tmp/pipe1'
pipe2 = '/tmp/pipe2'
try:
    os.mkfifo(pipe1, mode=0o777)

except OSError as oe:
    if oe.errno != errno.EEXIST:
        raise

try:
    os.mkfifo(pipe2, mode=0o777)
except OSError as oe:
    if oe.errno != errno.EEXIST:
        raise


# blocks here until writer opens read_pipe
read_pipe = open(pipe1, 'r')
write_pipe = open(pipe2, 'w')
print('opened pipes')

while True:
    r = read_pipe.readline().strip('\n')
    print(f'received {r}')
    if not r:
        print("Sender terminated")
        break
    write_pipe.write(f"Response: {r}\n")
    write_pipe.flush()
    print("wrote response")

C++ process:
#include <fstream>
#include <iostream>
#include <unistd.h>

int main() {
    std::string pipe1 = "/tmp/pipe1";
    std::string pipe2 = "/tmp/pipe2";

    int writes = 10;
    std::ofstream write_pipe(pipe1, std::ios::out);
    std::ifstream read_pipe(pipe2, std::ios::in);
    
    std::cout << "Opened pipes" << std::endl;
    while (writes--) {
        write_pipe << "Some useful message" << std::endl;
        printf("sent message\n");

        printf("Reading response...\n");
        std::string response;
        std::getline(read_pipe, response);
        printf("Received response: %s\n", response.c_str());
    }
    write_pipe.close();
    read_pipe.close();

    return 0;
}

Python version of the C++ code that produces the correct behaviour. I use time to time a round trip of messages (time from message sent to reading response), and it's around 10^-4 seconds.
pipe1 = '/tmp/pipe1'
pipe2 = '/tmp/pipe2'

write_file = open(pipe1, "w")
read_file = open(pipe2, 'r')

writes = 10
while writes > 0:
    writes -= 1
    start_time = time.time()
    write_file.write("Test message" + "\n")
    write_file.flush()
    print("sent message")

    response = read_file.readline()
    end_time = time.time()
    print(f"response was {response}")

Answer: There are multiple problems with your solution. The primary problem is that both programs are opening /tmp/pipe1 for writing and /tmp/pipe2 for reading. Which means that you don't actually have a bi-directional communication channel. It also means the opens deadlock because a blocking open for writing will not return until a process opens the fifo for reading.
A more subtle problem is that std::getline() won't return until the fifo is closed because it isn't seekable. So even if all the other problems are fixed the use of std::getline() won't allow the C++ program to process each line and tell the Python program it has done so.
Finally, your two programs do not act as a producer/consumer pair that operate in lockstep. Notice that they both start by writing a message intended for the other program. Rather than one of the programs waiting for a message from its peer.