Repository: ansible-collections/community.general

Issue:
Module listen ports facts extend output
##### SUMMARY

Include more infos from netstat or ss to the output. 

Mainly: 
- state
- foreign address
- process

As raised in #4762

##### ISSUE TYPE
- Feature Pull Request

##### COMPONENT NAME
listen_portcs_facts

##### ADDITIONAL INFORMATION

Changes: 
- adding the parameter: '-a', to netstat and ss to include all connections.
- rework netstat to a more general solution. 

Open topics: 

- first time trying to contribute to a public repo, even so the pull request is not 100% done I wanted to get some initial feedback. 
- is the approach I took okay or should we maybe add a parameter to listen_ports_facts to contain more information, while keeping the format of the current solution as standard.
- docstring type, is there a convention for this repo, I didn't find that info
- Should I rework ss to have the same setup then network
- Testing:
  - What is the verbatim output I should add? 
  - How to test SS, I would need to change docker version to for example a redhat 7 machine, but `ansible-test integration listen_ports_facts --docker centos7 -v` fails for me with Gather Factes -> unreachable 
  - Testing Netstat: `ansible-test integration listen_ports_facts --docker -v` uses netstat is working successful
  with the warning: `WARNING: Unable to determine context for the following test targets, they will be run on the target host: listen_ports_facts`

- how to run pre-commit hook for my changes? 
`community/general/plugins/modules/system/listen_ports_facts.py`
- what is the verbatim output? 

 @gaetan-craft maybe you give me some initial feedback?




Discussion:
@PKehnel this PR contains the following merge commits:

* https://github.com/ansible-collections/community.general/commit/4931fc8bc68db86c30adcb1446cebbf5ae7f37ff

Please [rebase your branch](http://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html) to remove these commits.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: merge_commit_notify --->
The test `ansible-test sanity --test pylint` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/pylint.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:369:0: missing-final-newline: Final newline missing
```
<!-- job_id: 47842 -->
<!-- job_id: 27b3fa5194cfa32f7b9f9136f606e69e -->
<!-- job_id: None -->
The test `ansible-test sanity --test pep8` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/pep8.html)] failed with 2 errors:
```
plugins/modules/system/listen_ports_facts.py:180:52: E231: missing whitespace after ','
plugins/modules/system/listen_ports_facts.py:369:11: W292: no newline at end of file
```
<!-- job_id: 47842 -->
<!-- job_id: a9d4e18310d0ec1a24e2c9fb75960039 -->
<!-- job_id: None -->
The test `ansible-test sanity --test import --python 2.7` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/import.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:180:70: traceback: SyntaxError: invalid syntax
```
<!-- job_id: 47842 -->
<!-- job_id: f5a9bec2e966f17b85b6532900ca7a97 -->
<!-- job_id: None -->
The test `ansible-test sanity --test compile --python 2.7` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/compile.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:180:70: SyntaxError: protocol, recv_q, send_q, local_address,foreign_address, *rest = line.split()
```
<!-- job_id: 47842 -->
<!-- job_id: bd794847919227dea7abdd0ab48616e2 -->
<!-- job_id: None -->
The test `ansible-test sanity --test pep8` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/pep8.html)] failed with 2 errors:
```
plugins/modules/system/listen_ports_facts.py:180:52: E231: missing whitespace after ','
plugins/modules/system/listen_ports_facts.py:369:11: W292: no newline at end of file
```
<!-- job_id: 47842 -->
<!-- job_id: a9d4e18310d0ec1a24e2c9fb75960039 -->
<!-- job_id: None -->
The test `ansible-test sanity --test import --python 2.7` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/import.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:180:70: traceback: SyntaxError: invalid syntax
```
<!-- job_id: 47842 -->
<!-- job_id: f5a9bec2e966f17b85b6532900ca7a97 -->
<!-- job_id: None -->
The test `ansible-test sanity --test import --python 2.6` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/import.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:180:70: traceback: SyntaxError: invalid syntax
```
<!-- job_id: 47842 -->
<!-- job_id: 459f3d9ce59c9ca1f56b060afeac6d08 -->
<!-- job_id: None -->
The test `ansible-test sanity --test compile --python 2.7` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/compile.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:180:70: SyntaxError: protocol, recv_q, send_q, local_address,foreign_address, *rest = line.split()
```
<!-- job_id: 47842 -->
<!-- job_id: bd794847919227dea7abdd0ab48616e2 -->
<!-- job_id: None -->
The test `ansible-test sanity --test compile --python 2.6` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/compile.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:180:70: SyntaxError: protocol, recv_q, send_q, local_address,foreign_address, *rest = line.split()
```
<!-- job_id: 47842 -->
<!-- job_id: 3118b6152a998f9e7c134834cc02cbe5 -->
<!-- job_id: None -->
The test `ansible-test sanity --test pep8` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/pep8.html)] failed with 2 errors:
```
plugins/modules/system/listen_ports_facts.py:180:52: E231: missing whitespace after ','
plugins/modules/system/listen_ports_facts.py:369:11: W292: no newline at end of file
```
<!-- job_id: 47842 -->
<!-- job_id: a9d4e18310d0ec1a24e2c9fb75960039 -->
<!-- job_id: None -->
The test `ansible-test sanity --test import --python 2.7` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/import.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:180:70: traceback: SyntaxError: invalid syntax
```
<!-- job_id: 47842 -->
<!-- job_id: f5a9bec2e966f17b85b6532900ca7a97 -->
<!-- job_id: None -->
The test `ansible-test sanity --test compile --python 2.7` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/compile.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:180:70: SyntaxError: protocol, recv_q, send_q, local_address,foreign_address, *rest = line.split()
```
<!-- job_id: 47842 -->
<!-- job_id: bd794847919227dea7abdd0ab48616e2 -->
<!-- job_id: None -->
The test `ansible-test sanity --test pylint` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/pylint.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:369:0: missing-final-newline: Final newline missing
```
<!-- job_id: 47842 -->
<!-- job_id: 27b3fa5194cfa32f7b9f9136f606e69e -->
<!-- job_id: None -->
The test `ansible-test sanity --test pep8` [[explain](https://docs.ansible.com/ansible/devel/dev_guide/testing/sanity/pep8.html)] failed with 2 errors:
```
plugins/modules/system/listen_ports_facts.py:180:52: E231: missing whitespace after ','
plugins/modules/system/listen_ports_facts.py:369:11: W292: no newline at end of file
```
<!-- job_id: 47842 -->
<!-- job_id: 39083e56b9b90f5c28c9e41b55b1ff50 -->
<!-- job_id: None -->
The test `ansible-test sanity --test import --python 2.7` [[explain](https://docs.ansible.com/ansible/devel/dev_guide/testing/sanity/import.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:180:70: traceback: SyntaxError: invalid syntax
```
<!-- job_id: 47842 -->
<!-- job_id: abc6fb3a532135e90630290b09030377 -->
<!-- job_id: None -->
The test `ansible-test sanity --test import --python 2.6` [[explain](https://docs.ansible.com/ansible/devel/dev_guide/testing/sanity/import.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:180:70: traceback: SyntaxError: invalid syntax
```
<!-- job_id: 47842 -->
<!-- job_id: 5bec6cdc12235af4a5ccf2d7c6a04602 -->
<!-- job_id: None -->
The test `ansible-test sanity --test compile --python 2.7` [[explain](https://docs.ansible.com/ansible/devel/dev_guide/testing/sanity/compile.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:180:70: SyntaxError: protocol, recv_q, send_q, local_address,foreign_address, *rest = line.split()
```
<!-- job_id: 47842 -->
<!-- job_id: b3a9f20265b1d6a34172590c630e5dc1 -->
<!-- job_id: None -->
The test `ansible-test sanity --test compile --python 2.6` [[explain](https://docs.ansible.com/ansible/devel/dev_guide/testing/sanity/compile.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:180:70: SyntaxError: protocol, recv_q, send_q, local_address,foreign_address, *rest = line.split()
```
<!-- job_id: 47842 -->
<!-- job_id: c2c21a1af0faa9370d706d4e99de192d -->
<!-- job_id: None -->
The test `ansible-test sanity --test pylint` [[explain](https://docs.ansible.com/ansible/devel/dev_guide/testing/sanity/pylint.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:369:0: missing-final-newline: Final newline missing
```
<!-- job_id: 47842 -->
<!-- job_id: 092ee2451fcfcfc7a8cdcfc635edc828 -->
<!-- job_id: None -->
The test `ansible-test sanity --test pylint` [[explain](https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/pylint.html)] failed with 1 error:
```
plugins/modules/system/listen_ports_facts.py:369:0: missing-final-newline: Final newline missing
```
<!-- job_id: 47842 -->
<!-- job_id: 27b3fa5194cfa32f7b9f9136f606e69e -->
<!-- job_id: None -->

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: shippable_test_result --->
cc @ndavison
[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@moonrail and @gaetan-craft could one of you give me an initial review and some feedback on my idea / thoughts. 

- is the approach I took okay or should we maybe add a parameter to listen_ports_facts to contain all information, while keeping the basic format of the current solution as standard. ( I prefer that solution actually)

- What is the verbatim output I should add?

- Also will try to add some more documentation
ready_for_review
## Docs Build 📝

Thank you for contribution!✨

This PR has been merged and your docs changes will be incorporated when they are next published.
@PKehnel this PR contains the following merge commits:

* https://github.com/ansible-collections/community.general/commit/305de35d31e9eb12005bf836efb280fd694c83e0

Please [rebase your branch](http://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html) to remove these commits.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: merge_commit_notify --->
From my side the PR is ready now.

To be discussed:
1.  Changing the output from  `address` to `local_address` might cause problems to current projects using this module. But from a logical point of view it makes sense, using `local_address` and `foreign_address` instead of `address` and `foreign_address`. @felixfontein What is the normal way to go here? 
2. Is `include_non_listening` a decent parameter name? This is the new parameter I introduced. Open for better suggestions.
@ndavison, @moonrail or @gaetan-craft could one of you review my changes? 
@felixfontein how can I generate the docs locally, didn't find anything for that? Thanks for your help so far.

> 1. Changing the output from  `address` to `local_address` might cause problems to current projects using this module. But from a logical point of view it makes sense, using `local_address` and `foreign_address` instead of `address` and `foreign_address`. @felixfontein What is the normal way to go here?

Breaking changes are not accepted. The best way is to add an option which allows to chose the new behavior. The default value must result in the old behavior.

> 2. @felixfontein how can I generate the docs locally, didn't find anything for that? Thanks for your help so far.

The easiest way is to install [antsibull-docs](https://github.com/ansible-community/antsibull-docs/), use its `sphinx-init` subcommand to create a sphinx environment for building community.general docs from the installed version (that is coming from this repo), and then to build the sphinx page.
Status update: 

- run with `include_non_listening ` set to `false` (default value), returns the exact same output as right now 
  - undid some changes I did earlier (tcp6 to tcp, local_adress to address, iterating ` name, pid in pids`)
- run with `include_non_listening ` set to `true` results in:
  - Shows both listening and non-listening sockets (for TCP this means established connections).
  - Adds the return values `state` and `foreign_address` to the returned facts
  - test are updated to validate this behavior 
- docs are updated (thanks @felixfontein for all the feedback)
If nobody complains, I'll merge for the next release.

### Backport to stable-5: 💚 backport PR created

✅ Backport PR branch: `patchback/backports/stable-5/c273498a03dc70ed4b57f5de0a0eb0702a63153a/pr-4953`

Backported as https://github.com/ansible-collections/community.general/pull/5041



<sub><sup>
--
🤖 @patchback
I'm built with [octomachinery](https://octomachinery.dev) and
my source is open — https://github.com/sanitizers/patchback-github-app.
</sup></sub>


@PKehnel thanks for your contribution!

Resolving patch:
diff --git a/plugins/modules/system/listen_ports_facts.py b/plugins/modules/system/listen_ports_facts.py
--- a/plugins/modules/system/listen_ports_facts.py
+++ b/plugins/modules/system/listen_ports_facts.py
@@ -257,8 +257,6 @@ def netStatParse(raw):
             }
             if result not in results:
                 results.append(result)
-            else:
-                raise EnvironmentError('Could not get process information for the listening ports.')
     return results
 
 


Test patch: