Repository: osvenskan/posix_ipc (language: python)

Issue:
Support for M1 Macs?
When trying to use this module on macOS on an ARM machine, this test fails on the following line:
https://github.com/osvenskan/posix_ipc/blob/88c384a4d9de3c601a11020dd5fa1971a6ed789a/tests/test_memory.py#L315
With this message: `AssertionError:16384 != 4096`

This is more than likely due to these processors using a 16k page size rather than the more usual 4k. The Asahi Linux community has already written much about this.


Discussion / hints:
I think there might be a confusion here between FS block size and memory page size.
`os.statvfs` returns the file system block size in its first value (`f_bsize`). Shared memory, not being on any real file system on any disk, apparently has not much to do with that.
Not to mention the difference between the physical page size and the page size of the process... :|
See: https://docs.python.org/3/library/resource.html#resource.getpagesize

Either way, if the exact new value of `mem.size` is not that important, only that it's greater than zero (and _maybe_ that it's a multiple of 1024 or 4096), perhaps the test could be made more lenient.
Thanks for the bug report! I have an M1 Mac, so I'll be able to experiment with this myself. 
Interestingly, I'm unable to re-create this failure on my system (2021 M1 Pro, OS version 12.6). Can you tell me more about the system where you see this fail?

Also, can you copy/paste the output of this command?

```
python -c "import os; print(os.statvfs('.'))"
```
Actually we are trying to set up the given package with the NIX package manager on an M1 mac and the build f this package fails when it runs the tests. I will try to provide some additional data to reproduce it.
os.statvfs_result(f_bsize=1048576, f_frsize=4096, f_blocks=242837545, f_bfree=142983978, f_bavail=142983978, f_files=1425860975, f_ffree=1424391824, f_favail=1424391824, f_flag=0, f_namemax=255)
I have two observations to make. First, the Open Group Specification (https://pubs.opengroup.org/onlinepubs/9699919799/functions/ftruncate.html) says clearly --

> If _fildes_ refers to a shared memory object, _ftruncate()_ shall set the size of the shared memory object to length.

Therefore, if macOS sets the size of the shared memory segment to anything other than what the caller passes, it's violating the specification. Apparently that's happening, otherwise this test wouldn't fail.

Second, I'm not sure how I settled on `os.statvfs('.')[1]` as the best way to get the file system block size. That's the `f_frsize` element of the structure returned by `statvfs()`.

I have occasionally mined Python's source code for ideas on how to make code cross platform, and it wouldn't surprise me if I copied the `statvfs()` call from Python. I added that call in 2014 (https://github.com/osvenskan/posix_ipc/commit/46382e06cbca5a04cebceb73bbec987093754264) and in the commit notes I referred to it as "block size magic" which is evidence that I didn't understand it and copied it from elsewhere. Python 2.7 was the prominent Python at the time, and it uses `f_frsize` as the block size: https://github.com/python/cpython/blob/f4c03484da59049eb62a9bf7777b963e2267d187/Lib/multiprocessing/heap.py#L95

I probably copied it from there. 

The documentation varies for `f_bsize` and `f_frsize` (both members of the struct returned by `statvfs()`. 

#### OpenGroup

```
unsigned long f_bsize    File system block size. 
unsigned long f_frsize   Fundamental file system block size. 
fsblkcnt_t    f_blocks   Total number of blocks on file system in units of f_frsize. 
```

https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/

#### IBM

```
unsigned long f_bsize 	The file system block size.
unsigned long f_frsize 	The fundamental file system block size.
unsigned long f_blocks 	The total number of blocks on the file system in units of f_frsize.
```

https://www.ibm.com/docs/en/zos/2.3.0?topic=functions-statvfs-get-file-system-information

#### Linux

```
unsigned long  f_bsize;    /* Filesystem block size */
unsigned long  f_frsize;   /* Fragment size */
fsblkcnt_t     f_blocks;   /* Size of fs in f_frsize units */
```

https://www.man7.org/linux/man-pages/man3/statvfs.3.html

#### macOS

This doc is for iPhoneOS; I had a hard time finding anything else.

```
f_bsize    The preferred length of I/O requests for files on this
           file system.  (Corresponds to the f_iosize member of
           struct statfs.)

f_frsize   The size in bytes of the minimum unit of allocation on
           this file system.  (This corresponds to the f_bsize member
           of struct statfs.)
```

About `f_blocks`, the documentation says "...the members f_blocks, f_bavail, and f_bfree (all of type fsblkcnt_t) represent the respective allocation-block counts."

https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/fstatvfs.3.html


### Summary About `statvfs()`

OpenGroup, IBM, and Linux seem to agree that `f_frsize` is the fundamental block size in this struct. Linux calls it "fragment size" for some odd reason, but it nevertheless defines `f_blocks` in units of `f_frsize` which is in agreement with OpenGroup and IBM's doc.

Apple is the odd one out here, probably due to its BSD roots. Nevertheless, it _should_ be able to allocate a block of `f_frsize`. 


## Conclusion 

The test is doing the right thing by relying on `f_frsize`. The behavior of macOS in this case disregards the spec. I'll add a special case for macOS to allow the shared memory's size to be ≥ the requested size.






I also want to point out for anyone reading that despite the title of the ticket, `posix_ipc` works just fine on M1 macs except for one test failing due to macOS allocating a larger memory segment than the caller requested.
@torokati44 @rhornig @tabgab the `develop` branch contains a fixed version of the test if you want to try it out. 
Thank you so much, @osvenskan!
Would you be able to push a new release including this fix on PyPI in the near future?
It would make our work a lot easier.
Yes, I have a new release ready to go, but I was hoping one of you could test it out first since I can't re-create the problem on my Mac. Have you had a chance to test it out? Here's the change that should fix the failing test.

https://github.com/osvenskan/posix_ipc/commit/c505aeaf6d7d37d9d0142a4922d168583bf5dcf6
Yes, I think @rhornig will try it out sometime soon.

Fix patch:
diff --git a/.gitignore b/.gitignore
index 74ba26d..14995ec 100644
--- a/.gitignore
+++ b/.gitignore
@@ -5,9 +5,10 @@ dist/*
 build/*
 probe_results.h
 prober/foo
-notes.txt
+notes.md
 posix_ipc.egg-info/*
 test.c
 demo/*.o
 demo/premise
 demo/conclusion
+post_dist.py
diff --git a/README.md b/README.md
index 8a103ec..0e44ded 100644
--- a/README.md
+++ b/README.md
@@ -14,11 +14,11 @@ Version 1.0.5 was the last version of `posix_ipc` to support both Python 2.7 and
 
 `posix_ipc` is available on PyPI:
 
-`pip install posix-ipc`
+	pip install posix-ipc
 
 Or you can clone the project and install it using `setup.py`:
 
-`python setup.py install`
+	python setup.py install
 
 ## License
 
diff --git a/USAGE.md b/USAGE.md
index 3b5433a..b3d7ca7 100644
--- a/USAGE.md
+++ b/USAGE.md
@@ -4,13 +4,13 @@ The Python extension module `posix_ipc` gives Python access to POSIX interproces
 
 macOS/OS X and other Unix-y platforms (including Windows + [Cygwin 1.7](http://www.cygwin.com/)) provide partial (or partially broken) support. See [the platform notes below](#platform-notes) for more details.
 
-This extension allows Python to interact with non-Python apps via IPC. If you want IPC between Python apps, you're better off using the [`multiprocessing` module](https://docs.python.org/3/library/multiprocessing.html) or the `multiprocessing.shared_memory module`(https://docs.python.org/3/library/multiprocessing.shared_memory.html) from Python's standard library.
+This extension allows Python to interact with non-Python apps via IPC. If you want IPC between Python apps, you're better off using the [`multiprocessing` module](https://docs.python.org/3/library/multiprocessing.html) or the [`multiprocessing.shared_memory module`](https://docs.python.org/3/library/multiprocessing.shared_memory.html) from Python's standard library.
 
 This module works under Python ≥ 3.6. It is released under [a BSD license](LICENSE).
 
-You can **download [posix_ipc version 1.1.0](http://semanchuk.com/philip/posix_ipc/posix_ipc-1.1.0.tar.gz)** [[MD5 sum]](http://semanchuk.com/philip/posix_ipc/posix_ipc-1.1.0.md5.txt) [[SHA1 sum]](http://semanchuk.com/philip/posix_ipc/posix_ipc-1.1.0.sha1.txt) which contains the source code, setup.py, installation instructions, tests, and [sample code](###sample-code). The exact same [posix_ipc tarball is also available on PyPI](https://pypi.python.org/pypi/posix_ipc). You can also find [the `posix_ipc` source code on GitHub](https://github.com/osvenskan/posix_ipc/).
+You can **download [posix_ipc version 1.1.1](http://semanchuk.com/philip/posix_ipc/releases/posix_ipc-1.1.1.tar.gz)** ([MD5](http://semanchuk.com/philip/posix_ipc/releases/posix_ipc-1.1.1.md5.txt), [SHA1](http://semanchuk.com/philip/posix_ipc/releases/posix_ipc-1.1.1.sha1.txt)) which contains the source code, `setup.py`, installation instructions, tests, and [sample code](###sample-code). The exact same [posix_ipc tarball is also available on PyPI](https://pypi.python.org/pypi/posix_ipc), along with pip-installable wheels for some platforms. You can also find [the `posix_ipc` source code on GitHub](https://github.com/osvenskan/posix_ipc/).
 
-You might want to read [all of the changes in this version](https://github.com/christopolise/posix_ipc/blob/29-convert-doc-to-markdown/history.md) and about some [known bugs](#known-bugs).
+You might want to read [all of the changes in this version](history.md) and about some [known bugs](#known-bugs).
 
 Note that this module doesn't support unnamed (anonymous) POSIX semaphores.
 
@@ -415,7 +415,7 @@ I don't know of any bugs in this code, but FreeBSD users should check the platfo
 
 ## Support for Older Pythons
 
-If you need to support Python < 2.7, try [posix_ipc version 0.9.9](http://semanchuk.com/philip/posix_ipc/posix_ipc-0.9.9.tar.gz) [[MD5 sum]](http://semanchuk.com/philip/posix_ipc/posix_ipc-0.9.9.md5.txt) [[SHA1 sum]](http://semanchuk.com/philip/posix_ipc/posix_ipc-0.9.9.sha1.txt).
+If you need to support Python < 2.7, try [posix_ipc version 0.9.9](http://semanchuk.com/philip/posix_ipc/releases/posix_ipc-0.9.9.tar.gz) [[MD5 sum]](http://semanchuk.com/philip/posix_ipc/releases/posix_ipc-0.9.9.md5.txt) [[SHA1 sum]](http://semanchuk.com/philip/posix_ipc/releases/posix_ipc-0.9.9.sha1.txt).
 
 # Platform Notes
 
diff --git a/VERSION b/VERSION
index 1cc5f65..8cfbc90 100644
--- a/VERSION
+++ b/VERSION
@@ -1,1 +1,1 @@
-1.1.0
\ No newline at end of file
+1.1.1
\ No newline at end of file
diff --git a/history.md b/history.md
index d5cf61b..203cafc 100644
--- a/history.md
+++ b/history.md
@@ -4,7 +4,13 @@ This is the version history for the [posix_ipc module](https://github.com/osvens
 
 As of version 1.0.0, I consider this module complete. I will continue to support it and look for useful features to add, but right now I don't see any.
 
-- **Current – 1.1.0 (25 November 2022) –**
+- **Current – 1.1.1 (31 December 2022) –**
+
+    - Fixed a bug introduced in 1.1.0 where setup would fail on systems where [the default file system encoding is not UTF-8](https://github.com/osvenskan/posix_ipc/issues/40).
+    - Made message queue tests more conservative to avoid resource exhaustion that [could occur on a system with an atypical configuration](https://github.com/osvenskan/posix_ipc/issues/42).
+    - Fixed (again) [a bug related to shared memory size](https://github.com/osvenskan/posix_ipc/issues/35), this time for real! Thanks to Rudolf Hornig for the fix.
+
+- 1.1.0 (25 November 2022) –
     
     - Drop support for Python 2.7 and ≤ 3.6.
     - Converted doc to Markdown. Thanks to GreatBahram and christopolise.
diff --git a/post_dist.py b/post_dist.py
deleted file mode 100755
index d54555a..0000000
--- a/post_dist.py
+++ /dev/null
@@ -1,52 +0,0 @@
-#!/usr/bin/env python3
-
-# Python imports
-import time
-import hashlib
-import shutil
-import os
-
-RSS_TIMESTAMP_FORMAT = "%a, %d %b %Y %H:%M:%S GMT"
-
-VERSION = open("VERSION").read().strip()
-
-# Make a copy of the tarball for posterity
-tarball_name = "posix_ipc-%s.tar.gz" % VERSION
-shutil.copyfile(os.path.join("dist", tarball_name),
-                os.path.join("releases", tarball_name))
-
-tarball_name = "releases/posix_ipc-%s.tar.gz" % VERSION
-md5_name = "releases/posix_ipc-%s.md5.txt" % VERSION
-sha1_name = "releases/posix_ipc-%s.sha1.txt" % VERSION
-
-# Generate the md5 hash of the tarball
-with open(tarball_name, 'rb') as f:
-    s = f.read()
-
-md5 = hashlib.md5(s).hexdigest()
-sha1 = hashlib.sha1(s).hexdigest()
-
-open(md5_name, "w").write(md5)
-open(sha1_name, "w").write(sha1)
-
-print("md5 = " + md5)
-print("sha1 = " + sha1)
-
-
-# Print an RSS item suitable for pasting into rss.xml
-timestamp = time.strftime(RSS_TIMESTAMP_FORMAT, time.gmtime())
-
-print("""
-
-        <item>
-            <guid isPermaLink="false">%s</guid>
-            <title>posix_ipc %s Released</title>
-            <pubDate>%s</pubDate>
-            <link>http://semanchuk.com/philip/posix_ipc/</link>
-            <description>Version %s of posix_ipc has been released.
-            </description>
-        </item>
-
-""" % (VERSION, VERSION, timestamp, VERSION))
-
-print('git tag rel' + VERSION)
diff --git a/setup.py b/setup.py
index dd8e251..c1a6826 100644
--- a/setup.py
+++ b/setup.py
@@ -16,7 +16,7 @@
 
 name = "posix_ipc"
 description = "POSIX IPC primitives (semaphores, shared memory and message queues) for Python"
-with open("README.md") as f:
+with open("README.md", encoding='utf-8') as f:
     long_description = f.read().strip()
 author = "Philip Semanchuk"
 author_email = "philip@semanchuk.com"


Test patch:
diff --git a/tests/test_memory.py b/tests/test_memory.py
index f869d2c..4498a1b 100644
--- a/tests/test_memory.py
+++ b/tests/test_memory.py
@@ -15,14 +15,6 @@
 _IS_MACOS = "Darwin" in platform.uname()
 
 
-def _get_block_size():
-    """Return block size as reported by operating system"""
-    # I thought it would be a good idea to pass self.mem.name to os.statvfs in case that
-    # filesystem is different from the regular one, but I get 'No such file or directory'
-    # when I do so. This happens on macOS and Linux, didn't test elsewhere.
-    return os.statvfs('.')[1]
-
-
 class TestMemory(tests_base.Base):
     """Exercise the SharedMemory class"""
     # SIZE should be something that's not a power of 2 since that's more
@@ -112,20 +104,18 @@ def test_mmap_size(self):
         # e.g. when self.SIZE = 3333, the
         # mmapped file is also 3333 bytes.
         #
-        # OS X's mmapped files always have sizes that are mod 4096 which is
-        # probably block size.
+        # macOS's mmapped files always have sizes that are mod mmap.PAGESIZE.
         #
         # I haven't tested other operating systems.
         #
         # AFAICT the specification doesn't demand that the size has to match
         # exactly, so this code accepts either value as correct.
-        block_size = _get_block_size()
 
-        delta = self.SIZE % block_size
+        delta = self.SIZE % mmap.PAGESIZE
 
         if delta:
             # Round up to nearest block size
-            crude_size = (self.SIZE - delta) + block_size
+            crude_size = (self.SIZE - delta) + mmap.PAGESIZE
         else:
             crude_size = self.SIZE
 
@@ -239,7 +229,7 @@ def setUp(self):
         # Creating a segment that's twice the size of the block size seems the best option since
         # dividing it in half still leaves a segment size that's likely to be acceptable on
         # all platforms.
-        self.original_size = _get_block_size() * 2
+        self.original_size = mmap.PAGESIZE * 2
         self.mem = posix_ipc.SharedMemory(None, posix_ipc.O_CREX, size=self.original_size)
 
     def tearDown(self):
@@ -283,14 +273,9 @@ def test_ftruncate_increase(self):
         """exercise increasing the size of an existing segment from 0 via ftruncate()"""
         mem = posix_ipc.SharedMemory(None, posix_ipc.O_CREX)
         self.assertEqual(mem.size, 0)
-        new_size = _get_block_size()
+        new_size = mmap.PAGESIZE
         os.ftruncate(mem.fd, new_size)
-        if _IS_MACOS:
-            # macOS sometimes misbehaves a little and returns a memory segment greater than
-            # the requested size. See https://github.com/osvenskan/posix_ipc/issues/35
-            self.assertTrue(mem.size >= new_size)
-        else:
-            self.assertEqual(mem.size, new_size)
+        self.assertEqual(mem.size, new_size)
 
 
 if __name__ == '__main__':
diff --git a/tests/test_message_queues.py b/tests/test_message_queues.py
index 1732836..ab2b4f6 100644
--- a/tests/test_message_queues.py
+++ b/tests/test_message_queues.py
@@ -52,7 +52,11 @@ def threaded_notification_handler_rearm(test_case_instance):
 class MessageQueueTestBase(tests_base.Base):
     """base class for MessageQueue test classes"""
     def setUp(self):
-        self.mq = posix_ipc.MessageQueue(None, posix_ipc.O_CREX)
+        # Create a small queue to reduce the likelihood of resource exhaustion
+        # See https://github.com/osvenskan/posix_ipc/issues/42
+        self.mq = posix_ipc.MessageQueue(None, posix_ipc.O_CREX,
+                                         max_messages=10,
+                                         max_message_size=10)
 
     def tearDown(self):
         if self.mq:
@@ -126,8 +130,8 @@ def test_name_as_bytes(self):
         name = bytes(tests_base.make_name(), 'ASCII')
         mq = posix_ipc.MessageQueue(name, posix_ipc.O_CREX)
         self.assertEqual(name, bytes(mq.name, 'ASCII'))
-        mq.unlink()
         mq.close()
+        mq.unlink()
 
     # don't bother testing mode, it's ignored by the OS?
 
@@ -177,8 +181,7 @@ def test_write_flag_existing_queue(self):
 
     def test_kwargs(self):
         """ensure init accepts keyword args as advertised"""
-        # mode 0x180 = 0600. Octal is difficult to express in Python 2/3 compatible code.
-        mq = posix_ipc.MessageQueue(None, flags=posix_ipc.O_CREX, mode=0x180, max_messages=1,
+        mq = posix_ipc.MessageQueue(None, flags=posix_ipc.O_CREX, mode=0o0600, max_messages=1,
                                     max_message_size=256, read=True, write=True)
         mq.close()
         mq.unlink()