@(#)Driver.porting.notes 1.2 92/04/08

NOTICE: This file is under construction. DO NOT PUBLICIZE ITS EXISTENCE
without checking with Deborah Bennett (deborah@dobbs.eng) first.

This file is considered Sun Proprietary. Do not disseminate the
contents outside of Sun Microsystems.

Notes on porting your 4.x device driver to SunOS/SVR4 5.0 (jupiter)
===================================================================

Comments on this information should be directed to Deborah Bennett, 
deborah@eng. Please do not edit this file without asking first. Thanks.

The notes in this file will eventually be compiled into a supplement 
to the manual "Writing Device Drivers" (hereafter referred to as WDD).
There is an beta draft of the Jupiter WDD, part number 800-6502-06,
dated February 1992. Or you can contact Kennan Rossi (kennan@eng)
for any later documentation.
Late news - the porting appendix *did not* make it into the
Jupiter FCS documentation. Deborah Bennett (deborah@eng) is
writing a white paper covering driver porting.
She also has an 11-page presentation on drivers in Solaris 2.0
which can be given to customers.

There is a mail alias for discussions of porting drivers to the
sun DDI. The alias is ddi-owners@dgdo. Requests to be added should
be sent to ddi-owners-request@dobbs.eng.

Important Definitions:
	WDD: Writing Device Drivers manual
	Bootstick: The new boot/prom/kernel interface system.
	devfs: The new way of creating the links in /dev so
		you can access your device
	Jupiter: Solaris 2.0 or SunOS 5.0. The first customer
		release of SVR4 on Sun platforms
	Zeus: a previous (developer's) release. Sun4c only.
	Mars: a future 5.0 dot-release. The cutover release.


This guide has two parts:
	- Overview of changes which affect driver writers.
	  This includes some basic reasoning behind the
	  Sun DDI/DKI, new boot-stick, and devfs systems.
	- Step by step instructions, organized by driver
	  entry points, on what needs changing in your driver.
	  If you find info is missing, PLEASE send mail
	  to deborah@eng so that info can be put in this
	  doc.

#################################################################
##  Overview of Changes in Jupiter which affect Driver Writers ##
#################################################################

**** What is the Sun DDI/DKI? ****
	(References:
		gigo:/export/nse/docs/i-ddi/ddidki/*
			pretty badly out of date
		Jupiter WDD - 800-6502-**
		Section 9 manual pages on a 5.0 system near
		you - latest manpages are on plummet:/usr/src/man)
Why Invent a Sun DDI/DKI:
	- The DDI/DKI is a new name for the routines formally called
	"kernel support routines" in the old WDD, and for the
	"well-known" entry points in the cdevsw/bdevsw structs.
	The intent is to specify a set of interfaces for drivers
	which can specify a binary and source code interface
	specification for device drivers. If your driver only
	uses routines and structures described in the section 9
	manpages (DDI-compliant), then your driver is guaranteed
	to be binary compatible across Sun Solaris platforms
	with the same processor (like sun4c, sun4, sun4m, sun4d).
	- Sun extended the AT&T DDI/DKI to support multiple architectures,
	loadable drivers and modules, and many other things.


**** What's new about boot? ****
	(Reference: gigo:/export/nse/docs/i-sun4c/jupiterboot.ps
	 somewhat out of date, but still valuable)


**** What's new this new devfs? ****
	(References:
		- Device Driver Installation, Configuration and the
		Physical Namespace by Tom Allen, Bill Pittore and
		Joe Provino
		- 5.0 Boot and Configuration by Joe Provino
		latest versions of both on wizard:/callisto/docs/misc/techs)


**** ANSI C ****
	- Use "volatile" and "const" type specifiers where appropriate.
	Prototype all entry points.

**** Header Files  ****
	See Include Files below. The headers have been greatly
	reorganized.

**** Things you can't do anymore ****
	- Many architecture-specific features have been hidden from
	driver writers behind DDI interfaces. Specific examples are
	elements of the dev_info struct, caches, memory management
	units (mmu), page tables. If you have been using unadvertised
	interfaces, you may have to re-architect your driver to
	be DDI-compliant. If you continue to use unadvertised interfaces,
	you lose all the source and binary compatibility guarantees of
	the DDI.
	- You can no longer access the user-process area corresponding
	to your driver thread of control. After all, it could be
	running on a different processor on an MP machine.
	No poking around in the user struct or proc struct.


**** New Features  ****
	- most of kernel and all drivers are now loadable.

######################################################################
##  Step by Step instructions on what needs changing in your driver ##
######################################################################

Below, I describe a general strategy to convert your driver from
4.x to 5.0. It is not the only way.
The strategy is to convert as much as possible under 4.x before
ever compiling on 5.0. (Loadablility falls here). Then you
convert enough to get the driver to compile on the 5.0 system.
Then you comment out all the guts of all entry points except those
dealing with loading and unloading (_init, _fini, xx_identify,
xx_probe, and xx_attach). Getting the driver to load and unload
is at least half the total work involved for most drivers.
Once it loads and unloads, then you can debug each entry point
in turn until you're done.

Steps in converting your driver to SunOS/SVR4 5.0 (jupiter).

Step 0) Make it loadable.
-------------------------

   If your current driver is not loadable, 
   I suggest that you convert your driver to a loadable driver
   (if you can) under 4.x before starting the port to the DDI.
   This will familiarize you with the loadable driver concept.
   The loadable functionality under Jupiter is not that different
   from that under 4.x on the Sun4c. There is good information on loadable 
   drivers under 4.x in the manual "Writing SBus Device
   Drivers" (p/n 800-5323-05, part of the SBus developer's kit). 


Step 1) Get the driver to compile under 5.0.
-------------------------------------------

Inside of Sun Engineering,
There are two ways to do this. Either you can compile your driver
using the installed SPARCcompilers (I call this building native)
or you can activate an NSE environment read-only to use your
local execset compilers. By this point (after beta) you should
build native unless you are working on a driver which is tracking
a special environment for some reason (such as it's for a new
architecture which isn't netinstallable yet).
Outside of Sun Engineering, building native is your only choice.

**** MAKEFILE NOTES: ****

Building Native:

To compile the driver native on your jupiter system (jup_beta and later)

- Use these compile lines:

cc -D_KERNEL -I. -c example_one.c
cc -D_KERNEL -I. -c example_two.c
as -P -D_ASM -D_KERNEL -I. -o example_asm.o example_asm.s
ld -r -o example example_one.o example_two.o example_asm.o

NOTE: unlike under 4.x, you MUST do an "ld" step, even if your
driver has only one .o file. You cannot modload a .o file anymore.

Building under a read-only NSE activation:

- Pick the relevant NSE environment. If you don't know which one
  you need to use, you probably should be building native. (see above)

- activate your desired environment read-only:
	activate -r <my_env>

- cd to the directory where your source code is kept. Note that this
  directory should be outside of an NSE control point (that is, not
  under /proto or /usr/src). Do a "make" as usual.

**** INCLUDE FILES ****

	All the kernel header files are located in the sys directory now.
	New header files you will certainly need are:

<sys/devops.h>
<sys/ddi.h>		# Note that these two headers MUST appear last.
<sys/sunddi.h>		# in the kernel header includes.

	- the file <sys/vddrv.h> has changed name to <sys/modctl.h>.

	- For full DDI compliance, you are only allowed to include
	kernel header files which are listed in the synopsis sections
	of the section 9 manual pages. No other kernel header files
	are allowed.

**** STRUCTURE CHANGES ****
	- Add or change your xx_ops structure to be the new form.
	See the dev_ops(9s) manpage.
	If you have no devo_reset routine, use nulldev.
	- Define an xx_cb_ops structure. See the cb_ops(9s) manpage.
	If you have no devmap routine, use nodev. If you have no
	detach routine, use nulldev. Be sure to set the cb_flag
	field to ( D_NEW | D_MTSAFE ) if your driver is
	MT-safe.
	- Eliminate your local cdevsw or bdevsw structure.
	- The struct vdldrv has changed name to struct modldrv,
	and its form has also changed. See the modldrv(9s) manpage.
	Be sure to declare mod_driverops or mod_pseudodrvops as
	extern structs as well.
	- Add the new struct modlinkage. See the modlinkage(9s)
	manpage.
	- If you used to use the mb structures, they are gone.
	The elements of the mb_driver struct are (more or less)
	contained in the dev_ops struct. The elements which used to
	be in the mb_ctlr and mb_device structures are now opaque
	data in the dev_info struct. The information in your xx.conf
	file will cause some properties to be created for your driver
	when it is loaded.

**** OTHER COMPILE NOTES ****

	- If under 4.1 you used to use the construct
	#ident "<some SCCS stuff>";     <--- note trailing semicolon
	you must drop the semicolon now.
	If you were still using 
	static char SCCSid[] = "<some SCCS stuff>";
	you should change that to the #ident form, with no semicolon.
	- the type "addr_t" is gone, replaced with "caddr_t".
	- Just to get the driver to compile, you can stub out and
	just prototype the new routines needed in the dev_ops
	and cb_ops structures, such as xx_getinfo. You will need
	to write these routines before the driver has any chance
	of loading, though.


**** GOOD PRACTICES ****
	Prototype all your entry points in the ansi-c style as noted in the WDD.
	Use the new "volatile" type specifier to declare software structs
	which overlay actual hardware registers, or for specifying
	any other location which could be updated by an alternate
	thread of control. Also, use the new "const" type specifier
	for an object or array which is initialized, but then never
	changed. See any good Ansi-C manual for further details.

Step 2) Get the driver to load and unload under 5.0.
----------------------------------------------------

First a note about modload versus first-reference loading.
In 4.x, the only way to load a loadable driver was with the modload
utility. In 5.0, drivers can be automatically loaded when their
corresponding node in /devices is referenced. However, during
early driver porting, you may want to still use modload and modunload
until you are fairly sure that your driver loads and unload properly.
Then you can add your driver to the system using the add_drv utility.
Note that modload now requires a full pathname. See the modload(1m) manpage.

**** LOADABLE DRIVER SUPPORT ****
	These routines must be working for loading/unloading
	to work properly:
	_init
	_fini
	_info
	xx_identify
	xx_probe (for NSID devices only)
	xx_attach
	xx_detach (if your driver will be unloadable)
	xx_getinfo

	You should be able to stub out the rest of your driver
	routines while debugging the loading/unloading.
	Once the driver loads and unloads, most of the work will
	be done for simple drivers.

- A preliminary note: the kernel log() function and kernel printf()
have been replaced by cmn_err(). See the cmn_err(9f) manpage.
Note that CE_PANIC will panic the system after printing your
message, and that some levels of cmn_err generate automatic
labels before your message, and newlines after.

- The routine you might have had in 4.x called xx_init, or xx_vdcmd,
which had three sections labelled VDLOAD, VDUNLOAD and VDSTAT,
has now been (approximately only) replaced with the three
entry points, _init, _fini, and _info.

_init:			(see the _fini(9e) manpage)
	- This routine should do any per-driver (rather than per-unit)
	initialization, such as allocating per-driver memory. 
	I believe that this routine is called before xx_identify(),
	xx_probe(), and xx_attach(), so don't assume anything about unit
	numbers or unit state structures being present.
	Anything which used to be in the VDLOAD case of xx_init
	belongs here, except that you no longer need to call vd_installmod.
	- I suggest that you call ddi_soft_state_init here.
	_init must call mod_install with the address of
	your modlinkage struct.

_fini:			(see the _fini(9e) manpage)
	- This routine should do any per-driver shutdown. Basically it
	should do quiesce all the units controlled by the driver, then
	free any memory allocated in _init. This routine will be called
	just before the driver executable image is removed from memory.
	Anything which used to be in the VDUNLOAD case of init_module
	belongs here, except that you no longer need to call vd_removemod.
	If the decommissioning is sucessful, call mod_remove with
	your modlinkage struct. Otherwise return an appropriate errno.

_info			(see the _fini(9e) manpage)
	- This routine is passed a modinfop *. You should return
	the return value from mod_info thusly:

	return (mod_info(&modlinkage, modinfop));

	You can also print out information about your driver.
	You may find the __TIME__ and __DATE__ preprocessor
	directives handy to record the compile date of the driver.


xx_info or xx_getinfo		(see the getinfo(9e) manpage)
	This new routine should return a dev_info pointer for the
	dev_t (DDI_INFO_DEVT2DEVINFO) or the instance (unit) corresponding
	to the dev_t (DDI_INFO_DEVT2INSTANCE).

- xx_identify()		(see the identify(9e) manpage)
	- The "name" is no longer passed to you, you must use
	ddi_get_name() to get it.
	- IMPORTANT: the unit number counting is now handled by the
	system framework. To get the unit number in any routine,
	call ddi_get_instance(dip). DO NOT COUNT UNITS ANYWHERE!
	Consider this another good reason to use the new
	ddi_soft_state_* routines to manage your state structures.
	(They're internally MT-safe, too.)
	- You can no longer count on xx_identify() being called for all
	units before xx_attach() is ever called.
	- Return value should be DDI_IDENTIFIED or DDI_NOT_IDENTIFIED.

- xx_probe()		(see the probe(9e) manpage)
	- Used to determine whether the NSID (non-self-identifying device)
	which might be present is present, and if so, what version,
	configuration, etc it is.
	- If you used to statically allocate an array of controller
	state structures and use them this way:
		register struct xx_ctlr *c = &xx_ctlrs[ctlr];
	you can now do this:
		 c = (struct xx_ctlr *) kmem_zalloc (sizeof (*c), KM_SLEEP);
		 ddi_set_driver_private(dev, (caddr_t) c);
	You then use ddi_get_driver_private to get the ctlr pointer
	elsewhere in the driver. You should also store your
	xx_units structs in the dev_info corresponding to each unit
	in the equivalent way.
	- If your driver supports multiple types of devices, you should
	use ddi_prop_create(9f) to create descriptive properties
	which the other driver routines may use to determine what
	to do. You may create as many properties as appropriate
	for your device. Note that the information in your xx.conf
	file will be turned into properties in your dev_info struct
	by the module loading process automagically.
	- You must now use ddi_map_regs to map in your registers.

- xx_attach()		(see the attach(9e) manpage)
	- xx_attach arguments have changed.
	- As noted above under xx_identify, don't count units
	anywhere. Use ddi_get_instance(dip) to get your 
	unit (instance) number.
	- kmem_*alloc() now require a second (flags) argument.
	See the kmem_alloc(9f) and kmem_zalloc(9f) manpages.
	- The slaveslot() routine has been replaced with ddi_slaveonly(9f).
	- Use ddi_map_regs() to map in your registers where you used
	to use map_regs() or mapin().
	- If you need iopb memory, the call
		rmalloc(iopbmap, size)
	can be replaced with
		ddi_iopb_alloc(), then ddi_dma_addr_setup().
	You may also need to call ddi_dma_devalign().
	- You are no longer allowed to look at the dev_info struct
	contents. The dev_info_t is an opaque pointer. There are
	various DDI routines to acquire the property information which
	used to be accessed by looking in the dev_info.
	See the ddi_prop_create(9f) and ddi_prop_op(9f) manpages
	for more details on property management.
	- addintr is replaced by ddi_add_intr. See the ddi_add_intr(9f)
	manpage.
	- It is no longer necessary to notify the system that you
	are using DVMA with a call to something like adddma.
	- If you used to use the splN and splr routines, the multi-
	threading of the kernel now makes these routines obsolete.
	You should be using mutexes to block out threads of control.
	A full discussion of this is beyond the scope of a porting guide.
	See also the discussion on making your driver MT-safe, below.
	You should call mutex_init on your mutexes here in xx_attach.
	- SBus drivers which followed the example of the sample
	driver in the SBus WDD probably have an xx_poll() routine which
	determines which unit is interrupting, then calls xx_intr()
	with the unit number. This is no longer necessary. You can call
	ddi_add_intr(9f) for each instance, use xx_intr as the
	int_handler argument, and pass the unit number
	or pointer to the unit structure in the int_handler_arg argument.
	This is a more general approach, and should work without
	modification if your device ever exists on another bus,
	such as an autovectoring bus like VME.
	- You should call ddi_create_minor_node for each attach.
	This creates the minor node structure used by the devfs
	to make the nodes in /devices at boot -r time.
	Be sure to call ddi_remove_minor_node if the call fails.
	- xx_attach should be careful to undo any unit allocations if
	a unit fails to attach. Be very careful to watch
	for recursive mutex_enter calls here.
	- the peek and poke calls have been replace with ddi_peek
	and ddi_poke. There are ddi_peek* calls for all sizes.
	- You may have to re-architect your driver since the mb structures
	are gone. The info in your xx.conf file will be turned into
	properties in your dev_info structure by the module loading
	process. Use ddi_get*prop() to fetch these properties.




xx_detach:		(see the detach(9e) manpage)
	- This new entry point allows your driver to be automatically
	unloaded to save system resources. NOTE: It is up to the driver
	to >allow< unloading. If you say no, the system cannot forcibly
	unload your driver.
	- xx_detach should undo and deallocate all the actions and
	allocations done in xx_attach. Be very careful to watch
	for recursive mutex_enter calls here.
	- If you previously had a decommissioning subroutine which
	you called in the VDUNLOAD step of your init_module
	routine, you can probably rework it slightly and make
	it your xx_detach routine.

**** DEBUGGING HINTS FOR LOADING/UNLOADING ****
	- You can vary the value of the system global "moddebug"
	to see what's happening at module loading/unloading time.
	The possible values are in <sys/modctl.h>.
	- Unlike in 4.x, you can now symbolically debug with kadb
	and loadable drivers. Remember that your kernel symbols will
	not be mapped into the kernel at all times, only when your
	driver is loaded.


Step 3) Get the rest of the driver entry points to work under 5.0.

- xx_open()		(See the open(9e) manpage)
	- note that the first arg to open() ONLY is a dev_t *,
	not a dev_t.
	- it is good practice to verify that the "otyp" you
	are passed is OTYP_CHR for character devices,
	or OTYP_BLK for block devices. This avoids having your driver
	opened in a layered open unexpectedly. These defines are in
	<sys/open.h>.
	- You are guaranteed that the system will not call the open
	routine for a particular unit unless the identify, probe
	and attach routines for that unit have been called and have
	returned. But this is per-instance, not per-driver.

- xx_close()		(See the close(9e) manpage)
	- Note that reac() now has two more arguments.

- xx_read()		(See the open(9e) manpage)
	- If you used to check user privelages using suser(), you
	should now use driv_priv(credp) instead.
	- IMPORTANT: you can no longer call physio passing the
	address of a buffer you have statically allocated. You may pass
	"(struct buf *) 0" as the second argument, allowing physio
	to allocate a buffer from the pool. If you must change this,
	remember to squirrel away the address of the allocated buffer
	in xx_strategy, because you need it to call biodone in your 
	interrupt routine.
	Another strategy is to use getrbuf(9f) to allocate the buf struct,
	and freerbuf(9f) to free it.
	- If you are a DVMA device, and your dma engine has limited
	bits of resolution (such as the top 8 bits being registered
	rather than a counter), then you should set up a ddi_dma_lim_t
	structure describing these limits. Pass the address of
	this struct as the 5th argument to ddi_dma_buf_setup().
	See ddi_dma_lim(9s).
	- If you need to DVMA data larger than MAXPHYS, see the
	ddi_dma_movwin(9f) manpage for details on the new sliding
	DVMA window feature.
	- The sleep() and wakeup() functions are still around, but
	should not be called from an MT-safe driver. Use
	cv_wait(9f), cv_signal(9f) and cv_broadcast(9f) instead.

- xx_write()		(See the write(9e) manpage)
	- Same notes as for xx_read, above.

- xx_ioctl()		(See the ioctl(9e) manpage)
	- Note that ioctl() now has two more arguments.
	- If you used to define your ioctl cmd args this way:
	#define	 MYIOCTL	_IOR(m, 1, u_int)	<-- m is not in quotes
	you will have to surround the letter m with single quotes to
	get the #define to work. Or you can use the new definition
	method:		(left-shifted letter or-ed with number)
	#define DIOC            ('d'<<8)
	#define DKIOCGGEOM      (DIOC|2)
	Theoretically, the magic number you use (DIOC above) should 
	be allocated by AT&T.
	The cred_p can be used to check credentials on the call, and the 
	rval_p can be used to return a pointer to a return value
	which means something (as opposed to the old method of always
	getting zero back for success). See the ioctl(9e) manpage.
	- As pointed out in the example in the WDD, you
	can no longer dereference arg directly. You must use copyin()
	and copyout() to move the data around. Note that the buffer
	pointer argument to copyin() and copyout() must be cast
	to caddr_t. The cmds which were defined as _IOW will use copyin(),
	_IOR will use copyout(), and _IOWR will use both.

- xx_strategy()		(See the strategy(9e) manpage)
	- Note that xx_strategy should now return an int. The value
	is ignored, so return whatever you like. Zero is safe.
	- If you used to find your unit number from the buf struct
	this way [ unit_no = getminor(bp->b_dev); ] <-- uses b_dev field
	it must change to
		unit_no = getminor(bp->b_edev);	<-- uses b_edev field
	- If you were using mb_mapalloc before, you should change
	that to using ddi_dma_buf_setup. If no "arg" is necessary,
	use (caddr_t) 0. You will need to use ddi_dma_htoc()
	to convert the dma handle into a dma cookie, from which
	you can get the kernel virtual dvma address to plug into
	your DMA engine address register.
	- If you are a DVMA device, and your dma engine has limited
	bits of resolution (such as the top 8 bits being registered
	rather than a counter), then you should set up a ddi_dma_lim_t
	structure describing these limits. Pass the address of
	this struct as the 5th argument to ddi_dma_buf_setup().
	See ddi_dma_lim(9s).
	-You should define a new element of your unit struct to store
	the dma handle, since you will need it later to free the
	memory after the DMA is completed.
	- If you used to allocate your buffers uncached, you can
	now use ddi_dma_sync() whenever you must require cache
	consistency. You need not allocate your buffers uncached.
	- the semantics of timeout() and untimeout() have changed.
	timeout() returns an identifier, which must be saved to pass
	to untimeout(). This may require changes to your unit struct,
	if you stored pending timeout information in any other form
	than an int for each timeout there could be.
	- As stated above, save your buf address.

- xx_poll() and xx_intr()
	- Use ddi_dma_free() where you used to use mb_mapfree().
	- See the note above under xx_attach about eliminating
	xx_poll.
	- Make sure you are calling biodone with the new buffer
	address, not the old one, if you used to allocate your own
	buf struct.

Step 4) Other General Notes

**** Mutexes versus Interrupt Blocking ****

Mutexes versus interrupts:
	Calls to spl() and splx() NO LONGER DO ANYTHING!
	All critical sections should be surrounded by mutex_enter()/
	mutex_exit() pairs.

Making a driver MT-safe:
	- In your cb_ops struct, change your driver compatibility
	flag (last struct element) thusly:

	D_NEW | D_MTSAFE,       /* driver compatibility flag    */

	- define a mutex in your driver state (softc)structure
	something like this:

	mutex_t         my_mutex;              /* mutex for any op */

	You can define multiple mutexes for different things you
	want to protect if you like. See cgthree.c for an example
	(it uses one mutex for the registers, another for the
	pixrect, and another for a busy flag). However, you can
	get by with just one.

	- In your attach routine, after the space for your mutex
	has been kmem_alloced, call mutex_init thusly:

	mutex_init(&my_units[unit_no]->my_mutex, "my mutex",
		MUTEX_DRIVER, (void *)0;

	- Wherever you want to protect something from access by
	anther thread, like a register or structure, protect that
	access by mutex_enter and mutex_exit calls. There are two 
	points to note. NEVER nest mutex_enter/mutex_exit pairs. (That is,
	never call mutex_enter again after calling it once.)
	Also, it used to be safe to assume that no other thread
	could access anything in your interrupt routine, since
	you were spl'd to your interrupt level. Now, another thread
	could get in there. So you will need mutex_enter and mutex_exit
	calls around critical regions in your interrupt routines
	as well. Also, be very careful of items you used to initialize
	on the basis that no thread of control could be in your
	xx_read and xx_write routines at the same time. Now there
	could be such threads.

	- IF you need to know if a mutex is held, do this:
	ASSERT(MUTEX_HELD(&my_structure->my_mutex));
	If the mutex is not held, the system will panic with an
	assertion failure.