# System Monitoring with Xymon/Other Docs/HOWTO/Devmon SNMP
## Devmon SNMP Hobbit Setup HOWTO

This howto is to explore processes of adding SNMP manager querying ability to a hobbit server. The assumptions and procedures are from http://www.techagent.com/devmon_snmp_hobbit_setup.htm with some minor editing.

## Assumptions

* Hobbit is installed and working.
* Devmon in installed with no apparent problems.
* You have a target device that has snmp enabled.
* You know the IP address of the target device.
* You know the read only (query) community string of the target device.
* No firewall is installed on the Linux machine or between your Linux machine and the snmp target device.
* You have net-snmp-utils (or equivalent) installed on the Linux machine.
* This procedure use a Windows XP machine as an example SNMP device. Instructions for installing/configuring an SNMP agent for a variety of operating systems are available.

## Initial setup and testing
### Ping the target snmp device

Do this from the hobbit server, of course. NOTE: In the examples below the snmp target device will be called winserver.
This tells us that the device is up and is reachable across the network.
If you can't ping the snmp device, you must troubleshoot this problem before going any further. Remember sometimes devices are configured to not respond to a ping.

### Try to read the sysDescr from the snmp device

This will tell us that we can reach the snmp device via snmp, we are using the correct read only (query) community string and we will use the returned information in future steps.
If you were successful, Hooray! This is a good thing:
* We can talk snmp to the device.
* We are using the correct community string.
* We are using the correct snmp version.
* We now know the sysDescr for the device, Devmon setup will need to know this.
If you weren't successful then you must troubleshoot this step before going any further. Perhaps try snmpget with the -d ( debug) option to gather more information and maybe try tcpdump to listen for any communication between the linux server and the snmp target device.

### Add the target device to your hobbit setup

Add a line for the device to your bb-hosts file. See the Devmon documentation for more information on the options you can add here.

## Adding an unrecognized device and/or new test
### Consider how Devmon operates

This was written by eschwim (the project admin). The way Devmon works is fairly simple.
* An outside process (most likely devmon running with the --readbbhosts flag) updates the Devmon database from the Hobbit or BigBrother bb-hosts file.
** In a single-node installation, the Devmon database is stored in the hosts.db file.
** In a multi-node installation, it is kept in a MySQL database.
* Devmon reads its templates.
** A single-node installation reads the templates from disk at the beginning of every polling cycle.
** The multi-node version reads the templates from the database, but only if they have been updated/changed since the last time it read them.
* Devmon does SNMP queries on all of the devices in its database. SNMP queries are optimized so that if the same SNMP OID is specified in multiple tests for a device, it is only queried once.
* Devmon applies template logic against the returned SNMP data. This involves doing transforms, applying thresholds, and then finally rendering the message to be sent to the display server.
* Devmon sends the rendered messages to the Display server.
* Devmon sleeps for any remaining time in the poll cycle.
* Usually, return to step 1. If the interval at which your external process updates your devmon database is not be the same as your poll interval, Devmon might actually go to step 2 instead.

### Create a template for the device

We now have a basic understanding of Devmon. We can talk to our target device via snmp, we know the sysDescr, and we have the device entered into our bb-hosts file. Since the device (a Windows 2000 server) is not one of the supplied templates, the next step is to create a new template using a preexisting one as a guide.
* In the templates directory, make a new directory for the new device.
* Copy a specs file from another device directory to use as a guide.
* Edit the specs file:
** Change the Vendor to a vendor name you want to use.
** Change the model to a model number you want to use.
** Change the snmpver to the snmpver of the device (you confirmed the version in an earlier step).
** Most importantly, change the sysdesc to match the sysDescr you received from the device earlier.
In this example, the full sysDescr was used, but you can use something shorter like "Software : Windows 2000 Version 5.0" or even "Windows 2000" as long as it's matched. It depends on how granular you want to be (e.g. treat x86 vs x64 devices differently).
Don't yet worry about a test directory like all the other devices have. We are just trying to make sure Devmon is going to work with hobbit.

### Confirm template matched

Run devmon with the --readbbhosts -vvv flags. When we do this the devmon process will read the bb-hosts file looking for the DEVMON tag. When it finds one, it will query the device for sysDescr and try to find a matching device in the templates directory's specs files. If a match is found then the device is added to the Devmon hosts.db. If devmon doesn't find a match then the device will be ignored.
If the discovery was unsuccessful (i.e. no Discovered line), you need to double check the sysDescr against the sysdesc in the specs file. Try copy/paste to avoid any transcription errors.

### Decide on a test for the device

We now know devmon has successfully identified the target device and has entered it into the hosts.db. Our next step is to add a test. You should pick a value from the MIB for the device that you want hobbit to check. On my test device I have loaded the free standard edition snmp package from http://www.snmp-informant.com/. This package allows me to check the free disk space on the machine using this oid (object identifier) .1.3.6.1.4.1.9600.1.1.1.1.5.2.67.58. So my test will be a check of free disk space. Let's test with snmpget first to confirm that this is a good oid:
As you can see a value was successfully retrieved, so this is going to be the basis of the test. Create its directory.

### Create a test: files

Now is the time we concern ourselves with the other files we skipped in the new directory we created earlier. There are 5 files that are used, the only ones required are oids, message, thresholds. The others are useful and you will probably end up using them, but our goal is just to get going.
* oids : This file contains the OIDs of the tests to be performed. Notice I just made up a name in the first field, this name will be used in the other files also, so you want to make it unique. The documentation that comes with Devmon is very good for these files, so I won't be going into great detail.
* message: This is used to construct the message that will go to the hobbit server. It uses the names you made up in the oids file. The win2000uptime_m is a variable created in the transforms file. Again the documentation is fine for this file.
* thresholds: This file sets your desired thresholds for the value you are watching. This is very self-explanatory. See the documentation.
Optional Files:
* transforms: This file gives you the ability to do some magic with the values retrieved from the snmp target. I've simply used it to divide the uptime to get minutes, but it seems very powerful. See the documentation.
* exceptions: This file is currently outside the scope of this howto. Sorry! See the documentation.

### Test!

Run devmon with the -f -p -vvvvvvvvv options, this will:
* keep devmon in the foreground
* not send the message to the hobbit server
* and be very verbose
Success, devmon is working, the snmp device is responding. Now just follow the documentation to setup devmon in cron.

## Debug example
### A Successful match

* The sysdesc is "SunOS snmpsolaris10 5.10 Generic_118833-36 sun4u"
* The devmon spec files for Solaris 10 created from compaq-server.
** "cp -rp templates/compaq-server templates/Solaris-5.10"
** Note: sysdesc filed in Solaris-5.10/specs file need to be 5.10 or SunOS. otherwise devmon won't be able to match it.
** Note: raid directory need to be exist, otherwise match will failed.
** modify Solaris-5.10/spec like following
* Matched log

### A mismatch

* installed templates from sourceforge that contains some cisco,apc,powerware and netap template files.
* following is an example for missing/wrong template file for a snmp agent on Solaris 10.
* Incorrect specs file under templates directory for snmpsolaris10

#### How to override a mismatch or no match

One can override the detected specs via the model tag in bb-hosts. For example, one could use DEVMON:model(dell;poweredge) for a Dell PowerEdge server running Microsoft Windows (which only reports "Hardware: x86 Family 15 Model 4 Stepping 3 AT/AT COMPATIBLE - Software: Windows Version 5.2 (Build 3790 Multiprocessor Free)" for Microsoft Windows Server 2003).

## Samples of SNMP system info on different devices
### NAS
#### ONStor 2260

Following is a system description of onstor 2260 device.

### Cisco 2950 with snmp agent turned on

* 2950 Manuals about snmp
* Enable snmp on cisco 2950 by telnet/ssh into cisco 2950.
** enable
** configure terminal
** snmp-server community public ro
** end
** show running-config
** copy running-config startup-config
* get sysDescr by snmpget command

### Solaris 10

* Solaris 10
** check if snmp package got installed.
* configure the snmp conf file.
* start/stop snmp agent.
* Running process

## Graphing

Some documentation for graphing is provided in the docs/GRAPHING file. There are two common ways to get graphs from Devmon data. If you are collecting data from an SNMP branch, you would use the instructions for "Graphing a Table". If you are collecting data from an SNMP leaf, you would use the instructions for "Graphing one or several Values".

### Graphing a Table

In the message file, one can add RRD definitions to a table definition. This is preferred if the oids are branches. The example given in the documentation is
This rrd tag does not belong on it's own line, rather the line defining the table to follow. This will add a comment to the generated HTML page with information that can be parsed by the devmon-rrd.pl script located in the extras directory under your devmon install. That will begin to generate the .rrd files for the table.
To parse the .rrd files to make a graph, you must create a definition for the graph in the hobbitgraph.cfg file in your Xymon install directory.
Then add the ,column=devmon setting to the TEST2RRD variable in hobbitserver.cfg.
Note that old versions of Devmon could only use exactly two datasets with specific names of ds0 and ds1.
Note that, if you add a table which will have multiple rows, you should add a --multigraphs setting to CGI_SVC_OPTS; see the FAQ (Why does my custom graph only show in the trends column but not its own column ?) for more information. If you don't do this, the graphs will only appear in the trends column and not in the expected column.

### Graphing one or several Values

One can format the message file as something hobbitd_rrd's "ncv" module will recognize. This is preferred if the oids are leaves. NCV stands for "Name Colon Value" and the "ncv" module will parse the generated HTML page for this data. In the message file within your devmon template, when you add a line to display the temperature of a UPS, you would add:
This would show the value on the web page, but also give the "ncv" module something to parse. It would set a variable of "Temperature" equal to whatever value {tempVar} happens to be, and add it to the .rrd with a DS name of "Temperature". After the .rrd files are being generated, you can add a definition to the hobbitgraph.cfg file that will read the .rrd and display the graph on the page.
To begin generating the .rrd files, one could specify an NCV setting in hobbitserver.cfg like the following for an APC UPS:
If one anticipates adding or removing values then it's better to use SPLITNCV instead of NCV, again for an APC UPS:
In both cases, add a ,column=ncv setting to the TEST2RRD variable in hobbitserver.cfg. Use only =ncv within the TEST2RRD variable, not =splitncv. See the hobbitd_rrd documentation for more information.

### Confirming data is available for a graph

The hobbitd_rrd daemons only read these values at initialization, so if Xymon is running, you'll need to kill the daemons or restart Xymon. Once either of these are in use, wait a few minutes for new data, then check the rrd directory for the usual data files.

### Output a graph from the data

If one or more rrd files are available, add the appropriate code to hobbitgraph.cfg. Then add ,column setting to the GRAPHS variable, in hobbitserver.cfg.
If there are multiple graphs, add ,column:column|column1|column2 or something similar to each host's TRENDS setting in bb-hosts. Note that you'll only get one graph in the normal column if it isn't a "multigraph" type, but many can appear in the trends column.

## References

* http://net-snmp.sourceforge.net/docs/README.solaris.html