hpcc-systems/HPCC-Platform — Issue #1332: reduce idle check frequency in dafilesrv
Author: rwagner42
Created: 2012-01-23 16:09:35+00:00
State: open

In our prod env, the thor400_92 has 4 nodes that have a skew on the disk space. Checking on them, it appears to be WARNINGs logged in the dafilesrv logs causing the disk space to be used up.

e.g. 10.173.92.2:/c$/DAFILESRV.10_05_11.log

WARNING: Inactive client(10.150.134.2:58837): last touch 1107979811 ms ago (1942988926, -1243998559) 6828079: /c$/thordata/thor_data400/key/infousa/abius/20080611c/autokey/phoneb2._2_of_401

It is using around ~200GB of dafilesrv logs on each node. I've confirmed it is occurring as recent as today. And it is only happening on:
10.173.92.2
10.173.92.63
10.173.92.121
10.173.92.163

Jake Smith: ok I see the warning in source
Jake Smith: suspect it will go away not so much because you're upgraded to new, but because you're restarted those dafilesrv's
Jake Smith: it may be a case of some too proactive logging
Jake Smith: what is 10.150.134.2 ?

Russell: The node's hostname identifies itself as "qaroxie134002"

Jake Smith: how much time between same log entries?

Russell: I'm seeing a lot all time on the same second, then 1 or 2 minutes goes by, then a lot all on the same second.

Jake Smith: but they may all be different (files or clients) but logged every minute or 2
Jake Smith: can u see time between same file+ip ?

Russell: Looking just at connections from 10.150.134.2, calling file /c$/thordata/thor_data400/key/taxpro/20080430f/autokey/namewords2._2_of_401 I'm seeing it get called 2 times a minutes, almost every minute. Sometimes it skips a minute.

Jake Smith: ok, I see code, looks like it checks for idle clients connections ~every minute - if it itslef idle
Jake Smith: by design
Jake Smith: could be useful info, but every min. is prob too much
Jake Smith: can u open an issue to reduce frequency in dafilesrv..