Repository: certtools/intelmq (Python)

Issue:
Reverse DNS bot returns same cached value for whole IPv4 subnet
We have found that the reverse DNS expert bot seems to return wrong answers. This is due to it caching responses, but also wrongly returning those cached values for other IP addresses in the same IPv4 /24 subnet.

For example, using ReverseDnsExpertBot with default settings (cache db 7, cache ttl 86400). The IP addresses and reverse DNS values are taken from the reverse_dns unit test, and are manually verified.
```
# intelmqctl run ReverseDns-Expert process -s -d -m '{"source.ip": "192.0.43.7", "extra.shouldbe": "icann.org"}'
Starting ReverseDns-Expert...
ReverseDns-Expert: ReverseDnsExpertBot initialized with id ReverseDns-Expert and intelmq 3.1.0 and python 3.9.16 (main, Mar 14 2023, 03:17:01) as process 321.
ReverseDns-Expert: Bot is starting.
ReverseDns-Expert: Bot initialization completed.
 * Message from cli will be used when processing.
 * Dryrun only, no message will be really sent through.
Processing...
[
    {
        "extra.shouldbe": "icann.org",
        "source.ip": "192.0.43.7",
        "source.reverse_dns": "icann.org"
    }
]
DRYRUN: Message would be sent now to '_default'!
DRYRUN: Message would be acknowledged now!
```
```
# intelmqctl run ReverseDns-Expert process -s -d -m '{"source.ip": "192.0.43.8", "extra.shouldbe": "43-8.any.icann.org"}'
[
    {
        "extra.shouldbe": "43-8.any.icann.org",
        "source.ip": "192.0.43.8",
        "source.reverse_dns": "icann.org"
    }
]
```
(extra unrelated output removed)
In the second run, a different source.ip value is used, but the cached value icann.org is returned by the bot The actual correct value should be 43-8.any.icann.org.
When the cache is cleared using redis-cli -n 7 FLUSHDB, the correct result is returned:

```
# intelmqctl run ReverseDns-Expert process -s -d -m '{"source.ip": "192.0.43.8", "extra.shouldbe": "43-8.any.icann.org"}'
[
    {
        "extra.shouldbe": "43-8.any.icann.org",
        "source.ip": "192.0.43.8",
        "source.reverse_dns": "43-8.any.icann.org"
    }
]
```

This seems to be caused by the bot using the first 24 bits of the IPv4 address (as a string of 0's and 1's) as the cache key. Therefore, IP addresses in the same /24 subnet yield the same cache key. For IPv6 the full 128 bits are used and is therefore not a problem. Why this bitstring is used as cache key, instead of the IP address (as a string) is a mystery for us.

Funny enough, this would have been caught in the unit test, as in the first EXAMPLE_INPUT, a source and destination IP address in the same /24 are set. However, in the output, the destination reverse_dns is commented out.



Discussion:
Thank you for finding and diagnosing this bug!

This bug dates back to the beginning of this bot in 2015 (0c46999a18). It's very interesting that no one else yet noticed it. I guess it was indeed intentional in the beginning, although entirely undocumented.

I create PR#2395 to fix this issue by generating cache keys from the complete address.



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d6ec6a9f36..2f5e94e459 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,41 @@
 CHANGELOG
 ==========
 
+3.2.1 (unreleased)
+------------------
+
+### Configuration
+
+### Core
+
+### Development
+
+### Data Format
+
+### Bots
+#### Collectors
+
+#### Parsers
+
+#### Experts
+- `intelmq.bots.experts.reverse_dns.expert`:
+  - Fix the cache key to not cache results for /24 (IPv4) and /128 (IPv6) networks but for single IP-Adresses (PR#2395 by Sebastian Wagner, fixes #2394).
+
+#### Outputs
+
+### Documentation
+
+### Packaging
+
+### Tests
+
+### Tools
+
+### Contrib
+
+### Known issues
+
+
 3.2.0 (2023-07-18)
 ------------------
 
diff --git a/NEWS.md b/NEWS.md
index 624d0490ad..8aaa5916a1 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -10,6 +10,29 @@ This file lists all changes which have an affect on the administration of IntelM
 Please refer to the change log for a full list of changes.
 
 
+3.2.1 Bugfix release (unreleased)
+---------------------------------
+
+### Reverse DNS Expert
+Until IntelMQ version 3.2.0, the bot incorrectly cached and re-used results for /24 networks instead of single IP addresses.
+If the bot retrieved the PTR for `192.0.43.7`, it was cached for `192.0.43.0/24` and used for all IP addresses in this range, for example for `192.0.43.8`.
+IntelMQ version 3.2.1 fixes this issue.
+
+The bugfix will correctly increase the cache sizes and decrease the performance, as less (incorrect) data is re-used.
+
+### Requirements
+
+### Tools
+
+### Data Format
+
+### Configuration
+
+### Libraries
+
+### Postgres databases
+
+
 3.2.0 Feature release (2023-07-18)
 ----------------------------------
 
diff --git a/intelmq/bots/experts/reverse_dns/expert.py b/intelmq/bots/experts/reverse_dns/expert.py
index 058fa3f788..26587bbed1 100644
--- a/intelmq/bots/experts/reverse_dns/expert.py
+++ b/intelmq/bots/experts/reverse_dns/expert.py
@@ -15,8 +15,6 @@
 from intelmq.lib.mixins import CacheMixin
 from intelmq.lib.utils import resolve_dns
 
-MINIMUM_BGP_PREFIX_IPV4 = 24
-MINIMUM_BGP_PREFIX_IPV6 = 128
 DNS_EXCEPTION_VALUE = "__dns-exception"
 
 
@@ -48,16 +46,9 @@ def process(self):
                 continue
 
             ip = event.get(ip_key)
-            ip_version = IPAddress.version(ip)
             ip_integer = IPAddress.to_int(ip)
 
-            if ip_version == 4:
-                minimum = MINIMUM_BGP_PREFIX_IPV4
-
-            elif ip_version == 6:
-                minimum = MINIMUM_BGP_PREFIX_IPV6
-
-            cache_key = bin(ip_integer)[2: minimum + 2]
+            cache_key = bin(ip_integer)[2:]
             cachevalue = self.cache_get(cache_key)
 
             result = None
diff --git a/intelmq/version.py b/intelmq/version.py
index 824b858af9..137bd12ed3 100644
--- a/intelmq/version.py
+++ b/intelmq/version.py
@@ -2,5 +2,5 @@
 #
 # SPDX-License-Identifier: AGPL-3.0-or-later
 
-__version_info__ = (3, 2, 0)
+__version_info__ = (3, 2, 1, 'a1')
 __version__ = '.'.join(map(str, __version_info__))


Test patch:
diff --git a/intelmq/tests/bots/experts/reverse_dns/test_expert.py b/intelmq/tests/bots/experts/reverse_dns/test_expert.py
index c7451226ce..66bb1d9064 100644
--- a/intelmq/tests/bots/experts/reverse_dns/test_expert.py
+++ b/intelmq/tests/bots/experts/reverse_dns/test_expert.py
@@ -17,20 +17,20 @@
 EXAMPLE_OUTPUT = {"__type": "Event",
                   "source.ip": "192.0.43.7",
                   "source.reverse_dns": "icann.org",
-                  "destination.ip": "192.0.43.8",
-                  "destination.reverse_dns": "icann.org",
-                  # manual verification shows another result:
-                  # "destination.reverse_dns": "43-8.any.icann.org.",       # pretty weird!
+                  "destination.ip": "192.0.43.8",  # in the same /24 as source.ip, certtools/intelmq#2394
+                  "destination.reverse_dns": "43-8.any.icann.org",
                   "time.observation": "2015-01-01T00:00:00+00:00",
                   }
 EXAMPLE_INPUT6 = {"__type": "Event",
                   "source.ip": "2001:500:88:200::8",  # iana.org
                   "source.reverse_dns": "example.com",
                   "time.observation": "2015-01-01T00:00:00+00:00",
+                  "destination.ip": "2001:500:88:200::7",  # has no reverse record, certtools/intelmq#2394
                   }
 EXAMPLE_OUTPUT6 = {"__type": "Event",
                    "source.ip": "2001:500:88:200::8",
                    "source.reverse_dns": "iana.org",
+                   "destination.ip": "2001:500:88:200::7",
                    "time.observation": "2015-01-01T00:00:00+00:00",
                    }
 INVALID_PTR_INP2 = {"__type": "Event",