# System Monitoring with Xymon/Other Docs/FAQ
## Introductory FAQ
### Q. What is hobbit ?

A. Hobbit was the old name used by Xymon System Monitoring tool. http://www.hswn.dk/hobbit/help/about.html

### Q. Why should I use Xymon instead of BB ?

* Speed. Hobbit runs much faster and is less resource intensive than Big Brother.
* More functionality built in
* GPL
* Integrated Trending
* A write up of why I use hobbit at http://gendalia.public.iastate.edu/Hobbit.txt

## User FAQ
### When is the next version going to be released ?

A. Hobbit is a FOSS project by hobbit developers who contribute to the project in their spare time. It is under active development but a release date for the next version has not been announced. The current development snapshot is available for download from sourceforge if required. If you are chasing a specific bug fix or feature not currently available within the existing production release, please search the archive and/or post a query to the general discussion mailing list to see if there is a solution available. You can subscribe to the list by sending an e-mail to hobbit-subscribe@hswn.dk. A searchable archive of the list is available at http://www.hswn.dk/hobbiton/
Please subscribe to the hobbit announcement list to be notified of the release of new versions. You can subscribe to the list by sending an e-mail to hobbit-announce-subscribe@hswn.dk.
There is another answer for this question at http://www.hswn.dk/hobbiton/2008/02/msg00227.html

### Where can I find more tests ?

A.
Deadcat http://www.deadcat.net Deadcat is mostly Big Brother scripts, but most of them will work on Hobbit without too much fuss.
The Shire Project (Xymonton) https://wiki.xymonton.org/doku.php Shire Project is specific to Xymon(Better than Deadcat)

### How do I put duplicate hosts in the bb-hosts file ?

A.
The first occurrence of the host is as normal. All further occurrences should be

### How does the conn test work if the first ping doesn't respond ?

A. bbtest-net calls either hobbitping or fping (configurable is FPING= in hobbitserver.cfg)

#### How many pings are sent simultaneously ?

A. Exactly how many parallel connections are being used depends on your operating system -

#### How many packets are sent each time ?

A. hobbitping normally stops pinging a host after receiving a single response, and uses that to determine the round-trip time.

#### What is the ping time-out ?

A The default timeout is 5 seconds.

#### Can we configure it?

A. Yes. The—timeout=N parameter to hobbitping allows you to set the timeout to N seconds.

### What do the conn test (ping) colours mean?

A. Green is an OK response, yellow with packet loss (not delays, just lost packets) and red when there is no response.

### How do I get the hobbit client to download etc/hobbitclient.cfg from the server (or other files in etc or ext), then restart?

A. Why? If the clients are set up to use server-side config, they don't even need the /etc/*.cfg files locally...

### Is there a way within Hobbit of creating a graph with data from multiple RRD files ?

A. If you want to create a graph from multiple rrds, I would suggest using the 3rd party tool called drraw http://web.taranis.org/drraw/

### How do I disable the REPEAT alerts ?

A. Leaving off the REPEAT= defaults to repeat alerts every 30 minutes. Setting REPEAT=0 defaults to every minute.

### The data being sent from my clients is being truncated.

A. The maximum message size can be set in hobbitserver.cfg

### Do I need to restart the Hobbit server after editing the config files ?

A. No. All files are reread every 5 minutes.

### Do I need to restart my Hobbit client after making a change ?

A. No. All client config files are reread periodically

### How do I monitor a web page ?

A. Add the following line to the bb-hosts file
or for https
Note that there are also content checks and post and browser options available. See the "HTTP TESTS" section of the bb-hosts(5) manual for details.

### My test only runs every hour. How do I get it to not go purple after 30 minutes ?

A. Sometimes you want something monitored/tested only once each hour (remote Internet site), once each day (backup job or DNS update), or even less often. Or perhaps more frequently (a critical connection once per minute or less).
There is a LIFETIME option when you call bb on the client to report status. It defines how long a status message is considered valid (i.e. fresh and not yet stale). Search "LIFETIME" on the bb man page for a bit more info. Syntax is like the following.
The defaults are 5 minute testing intervals and 30 minute status lifetimes, so 5 misses are accepted by default. The lifetime should be adjusted for the acceptable number of misses, otherwise false positives might occur. For example, if the host goes completely offline for 2 hours, or twice coincidentally overlapping each hourly reporting period, an hourly test probably doesn't need immediate debugging. Alternatively, for something being tested every minute, a lifetime of 30 minutes should probably be adjusted downward.
A couple of sample scenarios:
* If a script reports backup status only when a backup is finished, and the host or display is not communicative at that exact time, purple might or might not be desirable for that host's backup status in that NOC.
* If a script checks the size of each subdirectory within a parent directory, and the status is more intended for graphing than for green/yellow/red reporting, many misses may be acceptable.
Warning: The man page suggests "sligtly more than the interval between your tests". But if that was actually followed then the default timeout would be 6 minutes. It isn't 6, and it's not designed to be easily changed from 30: Instead of being an option in a configuration file, it's hardcoded in the hobbitd/hobbitd.c file in the handle_status function:

### Does Hobbit encrypt its transmissions ?

A. Natively, no. Some people have reported success using external encryption.

### How do I monitor files? I have added FILE /path/to/my/file to hobbitclient.cfg but still nothing happens.

A.
You also need to add monitored files to client-local.cfg (This tells the client to send the file metadata to the server) The server then uses the metadata and the config in hobbitclient.cfg to determine test results. Also make sure hobbit has read access to the files.

### How do I monitor log files other than the default ones?

Answer 1
Answer 2. I have added LOG /path/to/my/logfile WARNING COLOR=yellow to hobbitclient.cfg but still nothing happens.
You also need to add monitored log files to client-local.cfg. This tells the client to send the log file to the server because Hobbit messages protocol is bi-directional, not just hobbit client sending message to server. hobbit server can actually instruct hobbit client to send in log files that are not default ones.
An example client-local.cfg file.

### What's the meaning of the track alert-mail-number in the subject of hobbit alert emails ?

A. The number is the ack-code you can use for acknowledging the alert.

### How do I restrict access to Hobbit pages to specific people or groups ?

A. Apache has its own authentication. Use it.

### I am having a problem with devmon ....

A. Please post devmon related questions to the devmon support mailing list.

### How do I create custom scripts and graphs ?

A. http://xymonton.org/tutorials:customgraph

### Why does my custom graph only show in the trends column but not its own column ?

A. This is typical for tables. One must append the column name to the --multigraphs setting.
* Look in the source file web/hobbitsvc.c.
** Find the multigraphs assignment and note its value.
* Edit the configuration file server/etc/hobbitcgi.cfg:
** Find the CGI_SVC_OPTS assignment.
** Add a --multigraphs= value to the assigned string, including the columns that tend to have tabular data. For example: CGI_SVC_OPTS="--env=/home/hobbit/server/etc/hobbitserver.cfg --no-svcid --history=top --multigraphs=disk,if_load,if_dsc" (Note that, while the setting in hobbitsvc.c starts and ends with a comma, the setting in hobbitcgi.cfg does not.)
* Then try out the change (refresh the web page).
More details on --multigraphs is in the hobbitsvc.cgi documentation. (man hobbitsvc.cgi)

### I don't want to display column foo in my display. How do I do that ?

A. Add the entry NOCOLUMNS:foo,bar to hide column foo and bar.

### How do I check to ensure something is not running ?

A. In bb-hosts
This will cause the test to go red if FTP is running

### I want to monitor Windows servers. How do I do that ?

If you have to, use BBWin as your client. http://bbwin.sourceforge.net/

### What are the translations between BBWin's XML and Central Mode?

(Note: untested but hopefully accurate)
| BBWin approximation | hobbit-clients.cfg |
| <uptime> <setting name="delay" value="(bootlimit)" alarmcolor="yellow" /> <setting name="maxdelay" value="(toolonglimit)" alarmcolor="yellow" /> </uptime> | UP bootlimit toolonglimit |
| <cpu> <setting name="default" warnlevel="(warnlevel)" paniclevel="(paniclevel)" /> </cpu> | LOAD warnlevel paniclevel |
| N/A | CLOCK maximum-offset |
| <disk> <setting name="(drive)" warnlevel="(warn)" paniclevel="(panic)" /> <setting name="(drive)" ignore="true" /> </disk> | DISK drive warn panic DISK drive IGNORE |
| <memory> <setting name="physical" warnlevel="(warn)" paniclevel="(panic)" /> <setting name="virtual" warnlevel="(warn)" paniclevel="(panic)" /> <setting name="page" warnlevel="(warn)" paniclevel="(panic)" /> </memory> | MEMPHYS warn panic MEMACT warn panic MEMSWAP warn panic |
| <procs> <setting name="(processname)" rule="-=3" alarmcolor="(color)" comment="(text)" <setting name="(processname)" rule="+=4" alarmcolor="(color)" comment="(text)" </procs> | PROC processname 0 3 color TEXT=text PROC processname 4 -1 color TEXT=text |
| N/A | FILE filename color |
| (see fsmon.cfg) | DIR directory color SIZE<maxsize SIZE>minsize |
| <stats> </stats> | PORT |
| <svcs> <setting name="(svcname)" value="(status") autoreset="(startup)" alarmcolor="(color)"> </svcs> | SVC svcname startup status color |
| <who> </who> | N/A |
(Note: not completely accurate but good starting point)
| BBWin | client-local.cfg |
| <msgs> <match logfile="System"> <ignore logfile="System" value="(text string)"> </msgs> | eventlog:System ignore text string |

### I am having a problem with bbwin on my Windows......

A. Please post bbwin issues to the bbwin forum
The BBWIN client works well. I've re-used a script that funcioba before Big Brother. The result is left in C: \ BBWin \ logs.
Everything else is reported to the server but not external.
<?xml version="1.0" encoding="utf-8"?> <configuration>
</configuration>
File script generate out:
    OK
echo green ***Error en Chequeo SQLV, Servidor:%computername% >> C:\BBWin\logs\sqlv echo ^&green ***Error en Chequeo SQLV, Servidor:%computername% >> C:\BBWin\logs\sqlv
    BAD
echo red ***OK en Chequeo SQLV, Servidor:%computername% >> C:\BBWin\logs\sqlv echo ^&red ***OK en Chequeo SQLV, Servidor:%computername% >> C:\BBWin\logs\sqlv

### I set up my client, but nothing is appearing on the status page ?

A. Check the ghost client reports or the hobbitd status page. It could be misconfigured.

### We are changing the name of a host, and want to keep monitoring it, and keep the history. Is there a way ?

A. Check the Hobbit Tips & Trick page in help menu ~/server/bin/bb 127.0.0.1 "rename OLDHOSTNAME NEWHOSTNAME"

### I don't want to use rrdtool to create data averages and round-robin the data, I want to keep all data forever. Can I do it ?

A. There's a method of doing this in the current snapshot, including a new hobbitd_rrd manpage that describes how to run it,

### I don't have a compiler on my AIX system. Where can I get a precompiled Hobbit client ?

A. http://www.docum.org/twiki/bin/view/Hobbit/HobbitClients

### How do I unsubscribe from the Hobbit mailing list ?

A. If you must go, send an e-mail to hobbit-unsubscribe@hswn.dk

### How do I add a hobbit search engine plugin into Internet Explorer 7 ?

'A. Await contribution.

### How do I add a hobbit search engine plugin into FireFox 2 ?

A. Copy the following file and save it as "c:\Program Files\Mozilla Firefox\searchplugins\hobbit.xml" (this is the OpenSearch description format which is compatible both for FireFox 2 and Internet Explorer 7). Adjust the url address for your site accordingly.
Then restart Firefox. You should see a hobbit blue smile icon shown up in search bar. References:
* http://www.opensearch.org/Specifications/OpenSearch/1.1#OpenSearch_description_document
* http://developer.mozilla.org/en/docs/Creating_OpenSearch_plugins_for_Firefox

### Is it possible to disable and acknowledge alerts via email, how?

Include an IGNORE exception with hosts or alert that you wish to ignore:
or perhaps more specific:
and check configuration:

### Can I configure a maximum time limit a alert can be acknowledged for ?

A.

### Can I restrict what hosts can be disabled on the enable/disable page ?

A.

### Can I enable/disable on only one display server and have it show on both?

Typical installations with dual display servers have them configured to act independently. Each Xymon display server lists only itself in the XYMSERVERS setting in xymonserver.cfg, so usually, the enable/disable page only applies to the server that served the form.
However, the XYMSERVERS value can be overridden for each CGI script, and the enable/disable form will send updates to all Xymon servers defined in XYMSERVERS. To override for enable/disable, perform the following.
Create the file xymonserver-enadis.cfg containing:
Then edit cgioptions.cfg and add this line:
and modify CGI_ENADIS_OPTS to reference the new variable:

### Can the SMS alert format be reconfigured to display more or less information ?

A. Not directly, but an alert script (smsplus) has been written to provide more information than the default SMS output does. The script can be easily modified using the following environment variables (from the hobbit documentation):
| Name | Description |
| BBCOLORLEVEL | The current color of the status |
| BBALPHAMSG | The full text of the status log triggering the alert |
| ACKCODE | The "cookie" that can be used to acknowledge the alert |
| RCPT | The recipient, from the SCRIPT entry |
| BBHOSTNAME | The name of the host that the alert is about |
| MACHIP | The IP-address of the host that has a problem |
| BBSVCNAME | The name of the service that the alert is about |
| BBSVCNUM | The numeric code for the service. From SVCCODES definition. |
| BBHOSTSVC | HOSTNAME.SERVICE that the alert is about. |
| BBHOSTSVCCOMMAS | As BBHOSTSVC, but dots in the hostname replaced with commas |
| BBNUMERIC | A 22-digit number made by BBSVCNUM, MACHIP and ACKCODE. |
| RECOVERED | Is "1" if the service has recovered. |
| DOWNSECS | Number of seconds the service has been down. |
| DOWNSECSMSG | When recovered, holds the text "Event duration : N" where N is the DOWNSECS value. |

### Q. How do I enable SNMP monitoring with Hobbit Server?

A 1. http://cerebro.victoriacollege.edu/hobbit-trap.html
A 2. http://devmon.sourceforge.net

### Q. How to configure multiple yellow to red alert ?

A. Use the "badTEST" setting in bb-hosts (see the man-page). This delays a yellow or red status from appearing until it has stayed yellow (or red) for a number of test cycles. So you could use this to suppress the yellow status until it had been yellow for some time, so when it does turn yellow you know this is something you have to handle.
Or if your custom test reported a "red" status, you could make it go yellow for the first 5 test cycles, and red after that.
Henrik
Note: Currently, it only work for ping test.

### Q. How do I use hobbit client with BB server ?

A.

### Q. Can you use a hobbit server, but keep your bb clients?

A. Yes. Hobbit is 100% BB client compatible. but beware that after bb 1.9c, license become more restrictive, you need to pay for per-seat license.

### Q. What is Big Sister ?

** http://sourceforge.net/projects/bigsister/
** Big Sister is a port of an early version of Big Brother, written in Perl.

### Q. Is there a quick overview of System Monitoring Tools ?

* http://www.generalconcepts.com/resources/monitoring

### Q. What features should a monitoring system have ?

* Generic Monitoring System Features

### Q. Does NOCOLUMNS have to refer to a client line or can it refer to a page/subpage in bb-hosts ?

A. Waiting for contribution.

## Solaris Developer FAQ
### Warning messages from Sun Compiler

Followings are warning messages from using the Sun Compiler.

#### Q. "pointer to unsigned char "=" pointer to char"

* the error message.
* The source
* Note: getenv is documented as returning char *, not unsigned char *.

#### Others
## Administration FAQ
### Q. How do I enable RSS on hobbit server ?

A.
* In the hobbitserver.cfg file, change the BBGENOPTS variable from the following:
* To the following:
* The newly created RSS files in the www directory should resemble the following:

### Q. How do I configure SMF for hobbit on Solaris 10 ?

A. From: Everett, Vernon For those of you familiar with Solaris 10, you should know about services, but for some, adding new ones is a little tricky. To get Hobbit working as a service we need to do the following.
Create a file named /var/svc/manifest/application/hobbit.xml with the following content:
Take note of lines 37, 47 and 57, the lines that start "exec=". You may need to edit the path to your Hobbit start script.
To avoid confusion or possible issues, shut down your hobbit client at this point using the runclient script.
Now, as root, run the command
We should now have a service called hobbit.
(It will probably have gone online at this point)
You can now treat it as you would a regular service. If it hasn't gone online, kick it off as normal.
It may be necessary to do a disable and then an enable, but that should get it going.
And because we have set the default as enabled, the service should start automatically when you do a reboot.
Confirm it's all good by doing
All the usual scripts should be running.
If you don't want it as a service anymore, as root run
This will remove the service, and allow you to continue running it from the runclient script.

### Q. How do I configure Hobbit Client for Solaris 10 using SMF ?

A. copied from http://xymonton.org/addons:hobbitsmf

### Q. How do I configure Hobbit Server for Solaris 10 using SMF ?

A. copied from http://xymonton.org/addons:hobbitsmf

### Q. How do I enable devmon on hobbit server ?

A. This http://www.techagent.com/devmon_snmp_hobbit_setup.htm has some procedures to enable devmon on hobbit.

### Q. How do I remove a test ?

A.

### Q. How can I request the hobbit server to ask hobbit client to run a command locally based on an alert.

A.

### Q. How do I configure "GROUP" alerts ?

A. "GROUP" keyword is used in hobbit to classify many process name or disk partition name into different groups. This feature is needed in a big IT environment usually has different teams responsible for different areas of IT infrastructure. Ex. an IT organization usually consists of network,data/storage,backup, databases, application and Unix teams.
* hobbit-client.cfg : We need to specify which process will alert which GROUP.
* hobbit-alerts.cfg : In this file we then specify which email address receive the GROUP alert.

### Example

This is simple setup for learning purpose. Assuming when / is over 93% usage we want unix team to be paged.when /boot is over 15 percent apps team need to be alerted. Also cron process is dead, Unix team should be alerted. when Xvnc is over 120 processes then apps team need to be alerted also.

#### hobbit-alerts.cfg

The names after GROUP, have to be exact the same that are used in hobbit-client.cfg

### Debugging

Hobbit provide a very powerful debugging tool to trace the alert rules. From following example we check a host against all the rules in hobbit-alert.cfg.
* How do I debug "Oversize data/client msg" error message?
** following is an example error message
** Checking the size of msg file got sent from hobbit client side.
** Find out why msg.linux2.test.com.text is so big.
* Setup the debugging environment by running bbcmd. it will set up all the hobbit environment variables.
* Also have a look at command syntax of hobbitd_alert.
* Using the hobbitd_alert debugging command: A successful match, /boot disk usage has to be really over 16%.
* Look from "Match with" keywords to locate the exact rule got matched.
* Another check, Xvnc process limit has to be really triggered.

### Q. How do I get trimhistory to work for a hobbit server on Solaris ?

A. The default example in hobbit manpage is for hobbit server on Linux.
* run bbcmd first to inherit BBHIST etc. of variables.
* 31676645 is about 1 year of epoch time number.
* ./trimhistory—debug—cutoff=`/usr/bin/perl -e 'printf "%d\n", (time-31676645);'`
* References : http://solarisjedi.blogspot.com/2006/06/solaris-date-command-and-epoch-time.html

### Q. How do I exclude "info" and "trends" columns from the NK overview page?

A. As of Hobbitd 4.12 it is currently hard-coded that the "info" and "trends" columns show up on all pages, including the NK and BB2 pages. If you want those columns removed you'll have to edit the Hobbit source-code. The change is pretty simple. In the hobbit-4.1.2/bbdisplay/pagegen.c file, lines 121-123 look like this:
Change the "return 1" on both lines to "return 0", save the file, run "make" and either run "make install", or copy the bbdisplay/bbgen program to ~hobbit/server/bin/ . Next time the NK page is updated, those columns will be gone.

### Q. How do I use the internal HTTP test feature of Hobbit to test a Squid proxy server?

A. If you want to check that the service is actually functional use something like this in your bb-hosts file:

### Q. How do I use the internal HTTP test feature of Hobbit to test a proxy server with authentication to a Windows domain?

A. If you want to check that the service is actually functional use something like this in your bb-hosts file:

### Q. How do I compare graphs from different hosts on one page?

A. As of Hobbitd 4.12 there currently isn't a front-end to build the URL's needed for the graphs, but you can do it by hand for any graphs with -multi definitions (e.g. load average, swap):
* Find the base graph you want, e.g. the cpu "load average" graph for one of your hosts.
* In your browser, right-click the graph and select "view image" or "open image". You now have a view of your load graph only.
* In the address bar field you'll see the URL for this image. E.g.
Now, you can add more hosts after the "host=..." part of the URL - just list all of your hosts separated by commas. Like:

### Q. I just upgraded from the BigBrother client to the Hobbit client and I don't get any status for CPU or disk, but I get status for other tests

A. Most common cause is that the hostname used by the client is different from the hostname you have in your bb-hosts file. On the client, what's the hostname reported by the "uname -n" command ? If that is different from the hostname you have in the bb-hosts file, start the client with the "--hostname=THE.REAL.HOSTNAME" option.

### Q. How do I fix "Oversize status msg from 192.168.1.31 for test.my.com:ports truncated (n=508634, limit=262144)"

A.
Try to increase value of MAXMSG_STATUS in ~server/etc/hobbitserver.cfg :
limit=262144 is 256kB. You can divide n value for 1024 (508634/1024 = 496) then you can set MAXMSG_STATUS="500" and restart hobbit server.
B.
* We need to investigate why the hb client message is oversize. Like following we found we have msg.*.txt file that is over 512k. This is abonormal for bb message sampling of a system.
* Further investigation found that [ports] section of msg.*.txt has too many output from two "netstat -na" commands.

### Q. I'ld like to change the default period for graph display: by default hobbit displays trends with 4 graphs (48h,12d,48d,576d periods). It is possible change it to 48h,7d,30d,365d for example ?

A. It isn't configurable, but in the hobbit-4.2.0/web/hobbitgraph.c file near the top of the file you'll find these lines:
Change them to suit you. Then search that same file for the HOUR_GRAPH etc. further down; you'll find 1 place where each is used like this:
and you need to change that "persecs" calculation also for all 4 graph types.
Change the legend:
Then run "make" (from the hobbit-4.2.0 directory) and "make install" (or just copy the "web/hobbitgraph.cgi" file to your ~hobbit/server/bin/ directory).

## Graph FAQ
### Q. Why does my Hobbit server have the following http response graph ?

Why does the response time of the http service differ so much. Is this a mis-configured http server?
A.Hundreds of reasons could cause a delay on an http server. We see a graph differ only in a range between 2 and 10 ms. This is quite normal. Due to the automated scaling, the graph differences look more important or drastic than they actually are.

### Q. My Hobbit server has the following wrsmd* network traffic trending graph ?

A., Inbound traffic is much more than outbound, this is normal. The Hobbit server receives lots of bb/hb message from clients.

#### Q. What are those wrsmd* interfaces ?

A., The wrsmd(7D) (WCI Remote Shared Memory (WRSM) DLPI driver) status was reported by the Solaris 10 command "/usr/bin/kstat -p -s '[or]bytes64'" used for [ifstat] in hobbitclient-sunos.sh. I said "was" because on all our patched Solaris 10 servers, we have not seen this for quite a while.
You can avoid this output by using "/usr/bin/kstat -p -s '[or]bytes64' | grep -v wrsmd | sort" for [ifstat]. You need also to remove all ifstat.wrsmd*.rrd files.
Dominique UNIL - University of Lausanne

#### Q. Graph after correction

The wrsmd interfaces now disappeared from the graph.

### Q. How can we use Hobbit/BBWin client to collect Bit/s on the Network interfaces?

A., I'm very interesting in this question, but have not found an answer yet.

## Error FAQ
### semop failed, Invalid argument ?

A., Frequently seen when hobbit dies ungracefully. Stop Hobbit, and run,