msm: kgsl: add ftrace for cache operations

Add the event kgsl_mem_sync_cache. This event is
emitted when only a cache operation is actually
performed. Attempts to flush uncached memory,
which do nothing, do not cause this event.

Change-Id: Id4a940a6b50e08b54fbef0025c4b8aaa71641462
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
msm: kgsl: Add support for bulk cache operations

Add a new ioctl, IOCTL_KGSL_GPUMEM_SYNC_CACHE_BULK, which can be used
to sync a number of memory ids at once. This gives the driver an
opportunity to optimize the cache operations based on the total
working set of memory that needs to be managed.

Change-Id: I9693c54cb6f12468b7d9abb0afaef348e631a114
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
msm: kgsl: flush the entire cache when the bulk batch is large

On 8064 and 8974, flushing more than 16mb of virtual address
space is slower than flushing the entire cache. So flush
the entire cache when the working set is larger than this.
The threshold for full cache flush can be tuned at runtime via
the full_cache_threshold sysfs file.

Change-Id: If525e4c44eb043d0afc3fe42d7ef2c7de0ba2106
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
msm: kgsl: Add support for coresight

Adding support for Coresight debug bus to work with the GPU,
including registering graphics core with Coresight and a coresight
interface to GPU through sysfs.

Change-Id: I9508659ca7d7d67e8a8becba41d06be76360c570
Signed-off-by: Harsh Vardhan Dwivedi <hdwivedi@codeaurora.org>
msm: kgsl: Disable hardware clock gating for A330V2

Turn off automatic hardware clock gating for A330v2 GPUs and
revert to the hardware default values.

Change-Id: I55961e2b4c3d4ffc5739aa4ce78f2b5a29482851
Signed-off-by: Alex Wong <waiw@codeaurora.org>
Revert "msm: kgsl: Add support for coresight"

This reverts commit 139c3853e8bfae7af0168d984363ae4de7afa727.
msm: kgsl: implement server-side waits

	# This is the 1st commit message:

	msm: kgsl: embed kgsl_context struct in adreno_context struct

	Having a separate allocated struct for the device specific context
	makes ownership unclear, which could lead to reference counting
	problems or invalid pointers. Also, duplicate members were
	starting to appear in adreno_context because there wasn't a safe
	way to reach the kgsl_context from some parts of the adreno code.
	This can now be done via container_of().

	This change alters the lifecycle of the context->id, which is
	now freed when the context reference count hits zero rather
	than in kgsl_context_detach().

	It also changes the context creation and destruction sequence.
	The device specific code must allocate a structure containing
	a struct kgsl_context and passes a pointer it to kgsl_init_context()
	before doing any device specific initialization. There is also a
	separate drawctxt_detach() callback for doing device specific
	cleanup. This is separate from freeing memory, which is done
	by the drawctxt_destroy() callback.

	Change-Id: I7d238476a3bfec98fd8dbc28971cf3187a81dac2
	Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
	Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>

	# This is the 2nd commit message:

	msm: kgsl: Add a command dispatcher to manage the ringbuffer

	Implements a centralized dispatcher for sending user commands
	to the ringbuffer. Incoming commands are queued by context and
	sent to the hardware on a round robin basis ensuring each context
	a small burst of commands at a time.  Each command is tracked
	throughout the pipeline giving the dispatcher better knowledge
	of how the hardware is being used.  This will be the basis for
	future per-context and cross context enhancements as priority
	queuing and server-side syncronization.

	Change-Id: Ic0dedbad49a43e8e6096d1362829c800266c2de3
	Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>

	# This is the 3rd commit message:

	msm: kgsl: Only turn on the idle timer when active_cnt is 0

	Only turn on the idle timer when the GPU expected to be quiet.

	Change-Id: Ic0dedbad57846f1e7bf7820ec3152cd20598b448
	Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>

	# This is the 4th commit message:

	msm: kgsl: Add a ftrace event for active_cnt

	Add a new ftrace event for watching the rise and fall of active_cnt:

	  echo 1 > /sys/kernel/debug/tracing/events/kgsl/kgsl_active_count/enable

	This will give you the current active count and the caller of the function:

	  kgsl_active_count: d_name=kgsl-3d0 active_cnt=8e9 func=kgsl_ioctl

	Change-Id: Ic0dedbadc80019e96ce759d9d4e0ad43bbcfedd2
	Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>

	# This is the 5th commit message:

	msm: kgsl: Implement KGSL fault tolerance policy in the dispatcher

	Implement the KGSL fault tolerance policy for faults in the dispatcher.
	Replay (or skip) the inflight command batches as dictated by the policy,
	iterating progressively through the various behaviors.

	Change-Id: Ic0dedbade98cc3aa35b26813caf4265c74ccab56
	Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>

	# This is the 6th commit message:

	msm: kgsl: Don't process events if the timestamp hasn't changed

	Keep track of the global timestamp every time the event code runs.
	If the timestamp hasn't changed then we are caught up and we can
	politely bow out.  This avoids the situation where multiple
	interrupts queue the work queue multiple times:

	   IRQ
	     -> process events
	   IRQ
	   IRQ
	     -> process events

	The actual retired timestamp in the first work item might be well
	ahead of the delivered interrupts. The event loop will end up
	processing every event that has been retired by the hardware
	at that point. If the work item gets re-queued by a subesquent
	interrupt then we might have already addressed all the pending
	timestamps.

	Change-Id: Ic0dedbad79722654cb17e82b7149e93d3c3f86a0
	Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>

	# This is the 7th commit message:

	msm: kgsl: Make active_cnt an atomic variable

	In kgsl_active_cnt_light() the mutex was needed just to check and
	increment the active_cnt value.  Move active_cnt to an atomic to
	begin the task of freeing ourselves from the grip of the device
	mutex if we can avoid it.

	Change-Id: Ic0dedbad78e086e3aa3559fab8ecebc43539f769
	Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>

	# This is the 8th commit message:

	msm: kgsl: Add a new command submission API

	Add an new ioctl entry point for submitting commands to the GPU
	called IOCTL_KGSL_SUBMIT_COMMANDS.

	As with IOCTL_KGSL_RINGBUFFER_ISSUEIBCMDS the user passes a list of
	indirect buffers, flags and optionally a user specified timestamp. The
	old way of passing a list of indirect buffers is no longer supported.

	IOCTL_KGSL_SUBMIT_COMMANDS also allows the user to define a
	list of sync points for the command. Sync points are dependencies
	on events that need to be satisfied before the command will be issued
	to the hardware.  Events are designed to be flexible.  To start with
	the only events that are supported are GPU events for a given context/
	timestamp pair.

	Pending events are stored in a list in the command batch. As each event is
	expired it is deleted from the list. The adreno dispatcher won't send the
	command until the list is empty.  Sync points are not supported for Z180.

	CRs-Fixed: 468770
	Change-Id: Ic0dedbad5a5935f486acaeb033ae9a6010f82346
	Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>

	# This is the 9th commit message:

	msm: kgsl: add kgsl_sync_fence_waiter for server side sync

	For server side sync the KGSL kernel module needs to perform
	an asynchronous wait for a fence object prior to issuing
	subsequent commands.

	Change-Id: I1ee614aa3af84afc4813f1e47007f741beb3bc92
	Signed-off-by: Jeff Boody <jboody@codeaurora.org>
	Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>

	# This is the 10th commit message:

	msm: kgsl: Add support for KGSL_CMD_SYNCPOINT_TYPE_FENCE

	Allow command batches to wait for external fence sync events.

	Change-Id: Ic0dedbad3a211019e1cd3a3d62ab6a3e4d4eeb05
	Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>

	# This is the 11th commit message:

	msm: kgsl: fix potential double free of the kwaiter

	Change-Id: Ic0dedbad66a0af6eaef52b2ad53c067110bdc6e4
	Signed-off-by: Jeff Boody <jboody@codeaurora.org>
	Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>

	# This is the 12th commit message:

	msm: kgsl: free an event only after canceling successfully

	Change-Id: Ic0dedbade256443d090dd11df452dc9cdf65530b
	Signed-off-by: Jeff Boody <jboody@codeaurora.org>
	Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
iommu: msm: Don't treat address 0 as an error case

Currently, the iommu page table code treats a scattergather
list with physical address 0 as an error. This may not be
correct in all cases. Physical address 0 is a valid part
of the system and may be used for valid page allocations.
Nothing else in the system checks for physical address 0
for error so don't treat it as an error.

Change-Id: Ie9f0dae9dace4fff3b1c3449bc89c3afdd2e63a0
CRs-Fixed: 478304
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
msm: kgsl: Add the buffer user address to the debugfs memory list

For each buffer in the debugfs memory list print the address it is
mapped to in the user process.  This is useful for matching up the
user side allocations with what the kernel thinks it has.

Change-Id: Ic0dedbadd07a8f704c0bf73b183aaeeddbbdd129
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: Ling Wan <lingw@codeaurora.org>
msm: kgsl: Don't do intensive memory recovery when allocating big pages

We don't want to incur too much overhead when allocating big pages
so don't attempt to retry, perform reclaim, or run memory compaction
on high-order allocations.

Change-Id: Ic0dedbadb354c6faea34abec36aee268ac0f2c34
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: Ling Wan <lingw@codeaurora.org>
msm: kgsl: Switch to using the dmabuf APIs

The dmabuf APIs are now fully supported upstream. Since Ion
is now a dmabuf exporter, there is no need to call those APIs
directly. Switch to calling the dmabuf APIs to generate
scattergather lists for mapping to be more compatible.

Change-Id: Ie393e74bba080cba8a10e60908af7932729e6da6
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Ling Wan <lingw@codeaurora.org>
msm: kgsl: handle double free on timestamp

Make sure memory does not get freed twice if one of the
frees is free on timestamp.

Change-Id: Id03d69ae9b5cc598658794e65152a899b36fe314
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
Signed-off-by: Ling Wan <lingw@codeaurora.org>
msm: kgsl: Prevent race conditions when freeing memory

Multiple threads can call an ioctl to free a memory region.
Only one of these threads must be allowed to free memory and
the rest can return without freeing. Add a new pending flag
which is tested and set in a critical section guarded by
spinlock to prevent multiple ioctl threads from freeing the
same memory.

Also, a thread could be freeing a memory region that is
in use by another thread. Ensure that the detachment of the
memory from the process list and freeing of the memory always
happens in the same thread. This will prevent a situation where
the memory is being used by a thread which is detached from the
process list and is yet to be freed.

Seperate the selection of gpuaddr and mapping into pagetable. The
gpuaddr assignment needs to be done with process memory lock held
in a critical section but the mapping into pagetable can be done
separately without holding memory lock.

CRs-Fixed: 495144
Change-Id: Idf85fbd4bca29c18597f4b0e737c207f002ab266
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
Signed-off-by: Ling Wan <lingw@codeaurora.org>
msm: kgsl: Only reference the rb_node after taking spinlock

Derefernce the rb node after taking spinlock since the root
node can change and is supposed to be guarded by the spinlock

CRs-Fixed: 495129
Change-Id: I63de7755cf9d5fb548c082f43403000bcef5721b
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
Signed-off-by: Ling Wan <lingw@codeaurora.org>
msm: kgsl: Set alignment to 0 when low on mapped memory

If CPU map returns error due to low memory then we need to set
the alignment to 0 and instead of setting it to PAGE_OFFSET. This
is because we then simply search of a free memory gap at page
boundary and disrerard alignment requirement since memory is
low.

Change-Id: I828f5d0120d6d530e0e8857bcd96c9d5f266da10
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
Signed-off-by: Ling Wan <lingw@codeaurora.org>
msm: kgsl: Only initialize process structure once

If the debug root of the process structure is initialized
then it means that the remaining fields will also be
initialized. Check this field at the beginning of the
initialization routine and skip to end of the function. This
also by-passes the reinitialization of some fields because
of errorneous NULL pointer check of rb node. The rb node is
always initialized to NULL so checking if it is NULL to
determine whether the pointer is initialized or not is wrong.

CRs-Fixed: 498014
Change-Id: I73b2124c037187bc96942714dac62c0a72c26372
Signed-off-by: Shubhraprakash Das <sadas@codeaurora.org>
Signed-off-by: Ling Wan <lingw@codeaurora.org>
msm: kgsl: disable full cache flush if full_cache_threshold is 0

Make full_cache_threshold==0 mean never do a full cache flush,
since that is more useful than having it mean always do it.

Change-Id: I4ae46b6253f2d78a97f067b5f906637a3db1d5c8
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
Signed-off-by: Ling Wan <lingw@codeaurora.org>
msm: kgsl: pwrscale functions should be run while GPU is ACTIVE

Base the decision to call power scaling functions on the current
state, not the requested state.  The GPU may not transition to the
requested state; call through based on current GPU state instead.

Change-Id: I1524bc92c908dc50a60b5815f2221ae3feef59be
CRs-Fixed: 441847
Signed-off-by: Lucille Sylvester <lsylvest@codeaurora.org>
Signed-off-by: Ling Wan <lingw@codeaurora.org>
msm: kgsl: Bump up the GPU frequency for long batch processing

Compute workloads run without kernel interaction for long periods
of time.  If one is identified early, increase the frequency to
finish it as fast as possible, rather than waiting for the standard
algorithm to do so.

Change-Id: I213ccabfae5e1000cdc34bc1d92bdc3bad86383d
CRs-Fixed: 441847
Signed-off-by: Lucille Sylvester <lsylvest@codeaurora.org>
Signed-off-by: Ling Wan <lingw@codeaurora.org>
msm: kgsl: Restrict allocated memory alignment to a max of 2^31

The GPU memory allocation flags allow 8 bits of space to specify
a power of 2 aligment for the allocated memory block.  Cap the
actual alignment to 2^31 since any value more than that will have
unintended effects on a 32 bit system.

Change-Id: Ic0dedbadd42a9a886b8bfada0c7d415caaed65de
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: Ling Wan <lingw@codeaurora.org>