Fix TCP tx large payload and connection reuse.

net/tcp.s:
	net_tcp_sendbuf_flush: iterate over data if larger than payload.
	Somehow the sendbuf content to send is larger than it should be.

	fix reusing of tcp connections: the buffer's start/len fields are
	such that on reuse of a closed connection structure, only the first
	part of a file is sent.
	TODO: use buffer.s.

	TODO: minimize the buffer sizes - only keep data required for
	retransmission. On re-requesting the same page several times,
	memory usage increases until the VM crashes.
Minor printflag fix.
Add color arg to some debug macros
Exception handler fix

kernel/idt.s:
	don't load the opcode if EIP is 0 (as it will try negative offset).
	don't print scheduler task if there is no current task (pre scheduler
	enable).
	don't print SS0, SS1, SS2 as they are static. Print TSS descriptor
	type to inspect Busy flag.
Fix typo
Mutex/Sem for mem and fs.

kernel/mutex.s:
	add mutexes for MEM and FS.

	don't do 'hlt' in YIELD - causes #GP in nonprivileged mode.

	add LOCK_WRITE, LOCK_READ, UNLOCK_WRITE, and UNLOCK_READ macros,
	and extensive documentation as to why it is 'safe'/valid.

kernel/fs.s:
	add fs_handles_sem, and do (UN)LOCK_READ/WRITE whenever accessing
	the handles.

	add fs_open_, fs_stat_ and fs_handle_read_ unprivileged calls,
	which will call SEL_kernelCall to switch to kernel mode.

kernel/mem.s:
	prevent malloc reentrancy using a mutex.
Implement non-RING0 task scheduling.

kernel/schedule.s:
	Implement scheduling tasks with different privilege levels.

	Add TASK_FLAG_RINGx.

	scheduler_init:
		clear fs, gs, as they are currently unused and will
		cause #GP in CPL>0.

	schedule_near:
		when called with CPL>0 switch to kernelmode using the
		SEL_kernelMode callgate, and prepare the stack to look like an
		interrupt with privilege level change (eip,cs,eflags,esp,ss),
		then execute schedule_isr.

	schedule_isr:
		abandon 'lss esp', since the ss can be other than CPL0.
		The stack switch is the same for CPL0 and other tasks.

	scheduler_get_task:
		implement updating the current task for non-CPL0 tasks. The
		current stack will be from the TSS, which needs to be cleared
		for further use. The data on this stack is copied to the
		task stack, which is accessed in CPL0 initially, and will cause
		a privilege level change and a stack-switch in the IRET
		to the same stack (except higher up).

	schedule_task:
		implement TASK_FLAG_RING*, setting up cs, ds, es, ss using
		SEL_ring[0123](CS|DS).
		In case of non-CPL0 tasks, prepare the stack according to
		schedule_isr's expectations as described in scheduler_get_task.

kernel/gdt.s:
	add macro DEFTSS (same as DEFGDT for now).

	Add code/data selectors for ring 0,1,2,3, used by scheduler.

	Add a second tss - currently unused but initialized.

	Add callgate 'kernelMode' - similar to 'kernelCall' but does not
	do stack magic. It is used in schedule_near to do a privilege
	level change.

	Update macro GDT_STORE_SEG to preserve eax.o

	Update macro GDT_GET_BASE: mask out the CPL to get proper offsets in the GDT.

	Add macro GDT_GET_ACCESS to read the access byte.

	move macro PRINT_GDT from shell.s to gdt.s.

	rework kernel_callgate: add lots of debug, and set ds/es properly.

kernel/tss.s:
	fix the TSS structure: ESP1, ESP2 were missing.

	Add and initialize second TSS structure - at current unused.

kernel/shell.s:
	move PRINT_GDT macro to gdt.s.

	add shell command 'ascii' - print ascii table.

	cmd_exe: minor hack update in 'stealing' the filehandle buffer
	due to changes in fs.s.

kernel/hwdebug.s:
	disable DEBUG in memwrite_dword.

kernel/net/net.s:
	change the label for the network task to 'netq'.

kernel/debugger.s:
	add optional mutex argument to ASSERT_ARRAY_IDX
Run HTTPD in Ring1.

kernel/net/httpd.s:
	cmd_httpd: use TASK_FLAG_RING1 in scheduling the task.

	net_service_httpd_main: update calls to socket_(open|accept)
	and fs_(stat|open|handle_read) to their non-privileged versions.
	At current these are the same names with a _ suffix, that do
	a SEL_kernelCall callgate.

	wrap the fs_validate_handle_code to steal the buffer in a READ lock.

kernel/net/socket.s:
	add socket_(open|accept)_ non-privileged methods.

	update the tcp code in socket_close, socket_write, socket_flush
	to call the tcp functions (net_tcp_(sendbuf_flush|sendbuf|fin)
	in privileged mode if not running in CPL0.
Implement SHA-1

kernel/shell.s: add cmd_sha1 to print the hash for 'abc'.

lib/sha.s: SHA1 implementation.
Close parent dirs on file not found.
MAJOR: netq; sched; tcp; ssh; vmware/svgaII; mem;

This commit contains some disabled debug and some hacks, but is committed
due to the large amount of work aswell as to mark a point of relative stability.

	commit highlights:
		ADDS: vmware SVGA II Display driver
		ADDS: start of SSH daemon
		FIXES: seeming memory overwrites on ARRAY/buf_resize appearing
			in the scheduler and tcp, notably;
		IMPROVES: stresstest network stability using mutexes.
		BREAKS: network queue re-scheduling in some instances
		FIXES: use of UDP sockets.
		FIXES: TCP datastreams by using a receive buffer.
		FIXES: vmware TCLO channel communication.

DOC/Assembly.txt:
	fix doc on JG/JNLE: sign==overflow AND CF=0 (not OR!)

TODO:
	note some bugs that this commit fixes, but are not 100% verified.

kernel/Makefile:
	record vmware_svga2 driver.
	record lib/sha.s.

kernel/ata.s:
	add commented debug that'll print everytime a READ is done.

kernel/console.s:
	add debug for console struct ptr and keyboard struct ptr in cmd_consoles.
	Sometimes the virtual consoles cause a problem!

	Implement hardware cursor setting using legacy CRT ports - UNTESTED.

kernel/debug.s:
	allow DEBUG_DWORD [esp].

kernel/debugger.s:
	Add 'u': print memory handles.

kernel/defines.s:
	Reserve data sector for PCI_VID devices.
	TODO: merge with PCI_NIC.

kernel/dev.s:
	call vid_constructor for PCI class 3.0 (Display Adapter).

kernel/gfx.s:
	add realmode debugging:
		rm_dump_int_10: prints memory region pointed to by realmode
		IVT 0x6d (BIOS .....?)

		rm_trap_isr_register:
			register debugger trap ISR.

		rm_trap_isr:
			prints address/opcode for each executed instruction
			in order to trace the realmode VBE (Vesa BIOS extension).

		vesa_list_fb_modes:
			add debugging: optionally trap all instructions
			executed by the int 0x10 call using the eflags trap flag.

kernel/hash.s:
	buf_resize: add some code to zero out the new data.

kernel/idt.s:
	print minimal memory statistics on page fault to see whether there is
	more memory reserved than the paging tables account for.

kernel/kernel.s:
	add data and code section for PCI_VID;
	include vmware_svga2 driver.

	enable sipd;
	add sshd.

kernel/mem.s:
	give mem flags names and use those.

	print flag chars in print_handles instead of bitstring.

	add mem_find_handle$:
		similar to mem_find_handle$ but this is meant for external callers.

	mallocz:
		add DEBUG2; add disabled code to check if all data is indeed zeroed.

	malloc:
		some leftover debug.

	add mem_validate_handles:
		verify if next/prev 'fa' (free-by-address)
		linked list matches:
			prev.base+size == cur.base && cur.base+size==next.base.

	update get_handle_by_base$:
		add mutex fail debug, and DEBUG2: allow to trace decision path
		on handle splitting/merging.
		add another 'just in case' memset(0) .

	mrealloc:
		add debug.

	mreallocz:
		re-implement: do a mallocz, then copy the data. No handle merging.
		temporary solution!

		add mutex.

	mfree:
		add mutex.

		leftover debug.

	mdup:
		preserve ecx; remove jumps, since 'rep movsb' with ecx=0 has no
		effect.

kernel/mutex.s:
	Add a '(task/thread) local mutex' feature: a mutex is kept in
	.data tls_mutex.
	Add MUTEX_LOCAL_TEST, MUTEX_LOCAL_SETC and MUTEX_LOCAL_CLEARC.
	These are called in the MUTEX_LOCK/UNLOCK (disabled _LOCAL_MUTEX flag).

	MUTEX_SPINLOCK_:
		add a YIELD mechanism.
		Before, a bts was repeated until no carry, however,
		this would always succeed on the 2nd go since the carry bit was
		set regardles of outcome.
		This loop (still suffering from potential concurrency issues)
		will set the mutex, when it was already set it will clear it,
		then yield, and go again.

		seems dodgy to me though.

	MUTEX_SPINLOCK:
		update label names to higher value due to overlap elsewhere.

kernel/net/httpd.s:
	add (disabled) code to fork a client thread for each request.
	Disabled due to a problem with es register.

kernel/net/net.s:
	Major packet handling refactoring.

	add NET_QUEUE_DEBUG flag.

	record net/ssh.s

	fix net_print_ip_pair: sport/dport were swapped.

	net_rx_queue: re-implement as a circular array (FIFO) with packet dropping.
	(MAJOR!)
	The old method would reverse packet order sometimes.

	net_rx_packet: add debug, call net_rx_queue_newentry, and implement
	print dropped packet message on queue full.

	net_rx_queue_schedule: only schedule when task is not running.
	FIXME: _sometimes_ due to race condition the netq task keeps nto being
	scheduled, it seems.
	task is no longer scheduled with TASK_FLAG_RESCHEDULE, resulting in
	a short 'ps' list. (no dups!)

	net_rx_queue_handler: use new queue; add debug.

	net_rx_queue_print: MUTEX FIXME; print the packet queue/head/tail.

kernel/net/sip.s:
	UDP connections are now supported, so, reading from the listening socket
	is no longer supported. Instead, the socket is accepted and then the new
	socket is read.

	add request timeout response.

kernel/net/smtpd.s:
	print daemon started message.

kernel/net/socket.s:
	add debug.

	net_socket_deliver_udp: fix call to net_socket_in_append, which
	takes sock INDEX, not ptr.

kernel/net/tcp.s:
	use TCP_DEBUG_FLAGS (was duplicated code).

	Add tcp_conn_recv_buf.

	Add tcp_conn_tx_fin_seq to be able to know when the fin is acked.

	net_tcp_conn_get:
		Use MUTEX_SPINLOCK_ instead of MUTEX_LOCK loop,
		and MUTEX_UNLOCK_ instead of MUTEX_UNLOCK.

	net_tcp_conn_newentry:
		add mutex.

		add check for malloc going over 0x0040 0000.

		allocate receive buffer.

	net_tcp_conn_update:
		add mutex.

	net_tcp_conn_list:
		add mutex.
		print red 'B' when buffers are allocated and add debug to
		rint tx buffer info.

	net_tcp_ipv4_handle:
		on SYN, check if there is an ACK. If so, ignore the packet.
		port scanners use this approach, but it is invalid according
		to the spec.

		attempted to add mutex around net_sock_deliver_accept,
		but commented out. Also, remove recording of the socket
		in the tcp connection.

	net_tcp_handle:
		add mutex argument to the ASSERT_ARRAY_IDX macro call.

		add commented out code to check eax for -1 ater calling
		...update_ack, which would signify the connection is CLOSED
		due to having received the ACK for the TX FIN. However, this
		does not always apply.

		add mutex around tcp_connections update.

		set the STATE_FIN_ACK_TX using bts instead of or, as it allows
		for testing simutaneously. No apparent functional change here.

		add call to net_tcp_handle_payload regardless of PSH flag set,
		since not all packets that have a payload have this flag set.

	tcp_rx_sock: (proxy to net_socket_write)
		add mutex;
		call net_socket_find instead of relying on the socket being
		recorded in the tcp connection - which was disfunctional due
		to race conditions locking the entire network stack.

	add net_tcp_handle_payload$:
		append data to the receive buffer.

	net_tcp_handle_psh:
		add mutex.

		push data to the socket from the receive buffer, instead of
		the packet data having the PSH flag.

	net_tcp_conn_update_ack:
		add mutex.
		if FIN_TX flag is set, check whether the ack is for the FIN.
		If so, free the send buffer.
		Further, if FIN_ACK_RX and FIN_ACK_TX, return -1.

	net_tcp_sendbuf:
		add mutex.
		add debug for unusual high memory address.

	net_tcp_sendbuf_flush:
	net_tcp_fin:
		add mutex arg to ASSERT_ARRAY_IDX call.
		use new SPINLOCK_/UNLOCK_ mutexes

	net_tcp_send:
		fix bug where eax was assumed to still hold tcp_conn idx
		after being used to update the tcp packet flags and sending
		the packet.

kernel/paging.s:
	add [page_directory] using the ds relative address.

	paging_init: set CR4_PSE, which allows the PDE 'S' flag, meaning
	that the PDE refers to 4Mb and has no PTE.

	add paging_idmap_4m, to identity map a 4Mb physical page.
	This is needed for the vmware svga II driver whose frame and fifo
	buffers are beyond the paged tables, causing page faults.

	add paging_map_4k: nontested, nonfinished higher-resolution version
	of the paging_idmap_4m, which also allows to map physical addresses
	to different logical addresses.

	paging_show_usage: take the PDE 'S' into account as it lacks a PTE to print.

kernel/pci.s:
	add pci_get_bar method.

	add pci_get_bar_addr method, which masks out the config bits from
	the pci_get_bar result.

kernel/print.s:
	add nprint_ and nprintln_ which do not preserve esi and ecx nor
	stop when a 0 is encountered.

kernel/schedule.s:
	MAJOR REFACTORING.

	Change SCHEDULE_PRINT_FREQUENCY to PIT_FREQUENCY.

	scheduler_get_task$:
		ignore TASK_FLAG_SUSPENDED.
		have esi point to a pointer to 'ps' string when calling cmd_tasks.

	task_done:
		free the stackbuf (in case there is a burst of many tasks,
		a lot of memory will be kept reserved unnecessarily).

	task_queue_newentry:
		print error on malloc fail.

		clear the new schedule struct (safe now that stack is freed).
		See comments for kernel/mem.s:
		There were cases where memory pointers seemed to be overwritten
		with HTML content. After a lot of debugging, it turned out
		that the file loading code nor the http code were to blame,
		but instead, the memory allocated in buf_resize (through the
		ARRAY macros) returned uncleared memory. This was bad, since
		buffer pointers were kept to be reused.

		schedule_task: assume CPL0 if called from non-privileged code.

		leftover debug.

	schedule_print:
		call cmd_tasks as 'top'.

	cmd_tasks:
		when called as 'ps', behave differently then when called
		as 'top': for 'top', don't print the C(urrent) line,
		and don't print more than 20 lines.

	task_print_h$: preserve eax, esi.

kernel/vmware.s:
	nicer printing of RX / TX.

	vmware_chan_send:
		allow to send zero-length message (making ecx a param),
		which is used on the TCLO channel to signal a 'ping', and which is
		required before the VM will post messages on that channel.
		The result is that there is now better communication, PING works,
		and messages are received indicating 'unity mode' and such.

	add vmware_tclo_poll:
		sends empty message, then reads and parses the message if present.
		log all receiving messages over the rpcout channel.
		Respond 'OK ATR toolbox' on 'reset', which causes further messages
		to be received (the ATR toolbox is required).
		Respond 'OK ' to ping and DisplayTopology_Set (which is sent
		on entering Unity Mode).
		Respond 'ERROR Unknown command' to all other messages.

	vmd_vmcheck:
		use the tclo_poll for the tclo channel.
		Send different capabilities.

kernel/vmware_svga2.s:
	VMWare SVGA II Display Adapter Driver.

kernel/net/ssh.s:
	experimental beginnings of SSHD:
	accepts connections, sends the protocol identifier,
	receives client's protocol identifier,
	receives, parses and prints the KEX_INIT.

	implement HMAC SHA1.

lib/sha.s:
	sha1:
		properly preserve registers (sha_pad was called outside preservation).

	Add sha1_init, sha1_next, and sha1_finish:
		this allows to update the hash using multiple blocks.
		These methods require a context argument/state buffer of
		360 bytes to be maintained by the caller.
		sha1_next calls sha1_pad.

		Required for SSH MAC to prevent dupping data.

	TODO: the sha1_init and sha1_finish methods duplicate some code
	from the sha1 method.

root/www/index.html:
	note unwindable to bootloader.

	update device driver list.
	add file system list.
	add network protocol list.
Add base64_encode
Refactor PCI Device drivers.

Merge all driverinfo into one data section.
Add generic pci_find_driver 'constructor'.

Add baseclass constructor and api start/size info to pci structure.

add start of USB driver.

kernel/print.s:
	preserve al (eax) in nprint_.

kernel/vmware/:
	split vmware.s;
	implement VIX - partially.
PCI busmaster enable refactor

move am79... implementation of busmaster enabling to pci.s;

add dev_pci_busmaster_enable proxy to dev.s;

preserve edx in pci_(read|write)_config.
Fix base64 encode; implement decode
Add malloc_aligned; Add usb_ehci.

kernel/mem.s:
	add some methods to directly allocate aligned memory,
	leaving the alignment slackspace free for allocation.

kernel/usb.s:
	implement usb_ehci detection, interrupt handler and some status.
	Does not do anything other than print the port status when USB
	devices are plugged in or unplugged.
Net queue scheduled once.

The netq task is scheduled only once, and keeps processing the
queue, yielding when it is empty.

TODO: implement IO scheduling mutexes, where the scheduler does not
schedule a process unless some mutex is set.
It should also be possible to schedule the process regardless periodically
so it can do tcp connection cleanup.
Possible implementation is to set a timeout for rescheduling, which will
be reset by the net_rx_packet to indicate the netq should be scheduled ASAP.
vmw svga2: remove keypresses.