ClusterLabs/resource-agents — pull request #320 by kskmori (2013-10-02 11:03:26+00:00)
Title: IPaddr2/findif.sh: false failure detection

Note: This issue is still under investigation. The patch should be applied after we make sure that this is the right solution.

Description:
-----------------

IPaddr2 in 3.9.5 failed to monitor on occasion with the error:
```
Sep 12 15:41:06 node01 IPaddr2(prmIPaddr)[31269]: ERROR: Unknown interface [bond0] No such device.
Sep 12 15:41:06 node01 IPaddr2(prmIPaddr)[31269]: ERROR: [findif] failed
```

bond0 surely does exist on the system, and it runs fine most of the time, but it fails randomly; sometime an hour later or sometime a few days later, but we've seen it about 10 times so far.
There is no suspicious log messages in the syslog at that time.

The system had been running without any problem with resource-agents-3.9.2, and recently they have upgraded to resource-agents-3.9.5 for other reasons and the issue occured.
The distribution is RHEL5.7.

Obviously it is related to the parameter checking in findif.sh introduced in 3.9.5.

Diagnostics:
-----------------

ifcheck_ipv4() in findif.sh reads /proc/net/dev by using 'read' statement of bash and do loop to find the nic.

Our team is suspecting that it may fail to find an existing nic if the contents of /proc/net/dev has changed during the loop.

By looking at the source codes of bash and the kernel, it should work as below:

1. When /proc/net/dev is read, the kernel prepares the
contents onto the buffer based on the device information.
2. The 'read' statement of bash reads 128bytes each until it find a
newline, and remember the file position after the newline.
3. The next time 'read' is called, bash calls lseek() to the
remembered position - which makes the kernel to clear the
buffer and to regenerate new contents of the proc file.
4. As the results, lseek() may move to the wrong position and
bash may read an invalid line if the device information was
changed between 2. and 3.

It is definitly a problem if the size of the proc file is reduced during the 'read' loop because some lines may be missing.
That might be happen either when a nic device is removed or the statistics count is wrapped around and reset to 0.

I'm not confident yet if this scenario was really happening on the system in question - I could not see such evidence on the system so far - and now we are trying to reproduce it to collect the relavant infomation to see what is exactly happening.

Proposed patch:
-----------------

The patch is to use the ip command to check the interface instead of reading the /proc file. It is considered more reliable.

Our team is still working on investigating this issue.
I would appreciate if you have any hints or advices.